* mount: batched announcer + pooled peer conns for mount-to-mount RPCs
* peer_announcer.go: non-blocking EnqueueAnnounce + ticker flush that
groups fids by HRW owner, fans out one ChunkAnnounce per owner in
parallel. announcedAt is pruned at 2× TTL so it stays bounded.
* peer_dialer.go: PeerConnPool caches one grpc.ClientConn per peer
address; the announcer and (next PR) the fetcher share it so
steady-state owner RPCs skip the handshake cost entirely. Bounded
at 4096 cached entries; shutdown conns are transparently replaced.
* WFS starts both alongside the gRPC server; stops them on unmount.
* mount: wire tryPeerRead via FetchChunk streaming gRPC
Replaces the HTTP GET byte-transfer path with a gRPC server-stream
FetchChunk call. Same fall-through semantics: any failure drops
through to entryChunkGroup.ReadDataAt, so reads never slow below
status quo.
* peer_fetcher.go: tryPeerRead resolves the offset to a leaf chunk
(flattening manifests), asks the HRW owner for holders via
ChunkLookup, then opens FetchChunk on each holder in LRU order
(PR #5) until one succeeds. Assembled bytes are verified against
FileChunk.ETag end-to-end — the peer is still treated as
untrusted. Reuses the shared PeerConnPool from PR #6 for all
outbound gRPC.
* peer_grpc.go: expose SelfAddr() so the fetcher can avoid dialing
itself on a self-owned fid.
* filehandle_read.go: tryPeerRead slot between tryRDMARead and
entryChunkGroup.ReadDataAt. Gated by option.PeerEnabled and the
presence of peerGrpcServer (the single identity test).
Read ordering with the feature enabled is now:
local cache -> RDMA sidecar -> peer mount (gRPC stream) -> volume server
One port, one identity, one connection pool — no more HTTP bytecast.
* test(fuse_p2p): end-to-end CI test for peer chunk sharing
Adds a FUSE-backed integration test that proves mount B can satisfy a
read from mount A's chunk cache instead of the volume tier.
Layout (modelled on test/fuse_dlm):
test/fuse_p2p/framework_test.go — cluster harness (1 master,
1 volume, 1 filer, N mounts,
all with -peer.enable)
test/fuse_p2p/peer_chunk_sharing_test.go
— writer-reader scenario
The test (TestPeerChunkSharing_ReadersPullFromPeerCache):
1. Starts 3 mounts. Three is the sweet spot: with 2 mounts, HRW owner
of a chunk is self ~50 % of the time (peer path short-circuits);
with 3+ it drops to ≤ 1/3, so a multi-chunk file almost certainly
exercises the remote-owner fan-out.
2. Mount 0 writes a ~8 MiB file, then reads it back through its own
FUSE to warm its chunk cache.
3. Waits for seed convergence (one full MountList refresh) plus an
announcer flush cycle, so chunk-holder entries have reached each
HRW owner.
4. Mount 1 reads the same file.
5. Verifies byte-for-byte equality AND greps mount 1's log for
"peer read successful" — content matching alone is not proof
(the volume fallback would also succeed), so the log marker is
what distinguishes p2p from fallback.
Workflow .github/workflows/fuse-p2p-integration.yml triggers on any
change to mount/filer peer code, the p2p protos, or the test itself.
Failure artifacts (server + mount logs) are uploaded for 3 days.
Mounts run with -v=4 so the tryPeerRead success/failure glog messages
land in the log file the test greps.
mount: batched announcer + pooled peer conns for mount-to-mount RPCs
* peer_announcer.go: non-blocking EnqueueAnnounce + ticker flush that
groups fids by HRW owner, fans out one ChunkAnnounce per owner in
parallel. announcedAt is pruned at 2× TTL so it stays bounded.
* peer_dialer.go: PeerConnPool caches one grpc.ClientConn per peer
address; the announcer and (next PR) the fetcher share it so
steady-state owner RPCs skip the handshake cost entirely. Bounded
at 4096 cached entries; shutdown conns are transparently replaced.
* WFS starts both alongside the gRPC server; stops them on unmount.
mount: tier-2 chunk directory + FetchChunk streaming on one gRPC port
Collapses the old two-port design (HTTP peer-serve + separate gRPC
directory) into a single gRPC service that handles every mount-to-
mount exchange: ChunkAnnounce, ChunkLookup, and the new FetchChunk
byte stream.
* peer_directory.go: fid -> holders shard, HRW-gated; returns holders
in LRU order; capacity-bounded; Sweep handles eviction under
write-lock while Lookup runs under RLock (hot path is concurrent).
* peer_grpc.go: single MountPeer gRPC server implementing all three
RPCs. FetchChunk frames bytes at 1 MiB per Send so the default
4 MiB message cap does not constrain chunk size; cache miss
returns gRPC NOT_FOUND so clients distinguish miss from transport
error. Reuses pb.NewGrpcServer for consistent keepalive + msg-size
tuning.
* peer_bytepool.go: sync.Pool wrapper around *[]byte that the server
uses to avoid a fresh 8 MiB allocation per FetchChunk call.
* WFS wiring starts the gRPC server on option.PeerListen (the single
peer port) using the advertise address resolved in PR #3 as the
HRW identity. A background sweeper evicts expired directory
entries every 60 s.
* proto: define MountRegister/MountList and MountPeer service
Adds the wire types for peer chunk sharing between weed mount clients:
* filer.proto: MountRegister / MountList RPCs so each mount can heartbeat
its peer-serve address into a filer-hosted registry, and refresh the
list of peers. Tiny payload; the filer stores only O(fleet_size) state.
* mount_peer.proto (new): ChunkAnnounce / ChunkLookup RPCs for the
mount-to-mount chunk directory. Each fid's directory entry lives on
an HRW-assigned mount; announces and lookups route to that mount.
No behavior yet — later PRs wire the RPCs into the filer and mount.
See design-weed-mount-peer-chunk-sharing.md for the full design.
* filer: add mount-server registry behind -peer.registry.enable
Implements tier 1 of the peer chunk sharing design: an in-memory registry
of live weed mount servers, keyed by peer address, refreshed by
MountRegister heartbeats and served by MountList.
* weed/filer/peer_registry.go: thread-safe map with TTL eviction; lazy
sweep on List plus a background sweeper goroutine for bounded memory.
* weed/server/filer_grpc_server_peer.go: MountRegister / MountList RPC
handlers. When -peer.registry.enable is false (the default), both RPCs
are silent no-ops so probing older filers is harmless.
* -peer.registry.enable flag on weed filer; FilerOption.PeerRegistryEnabled
wires it through.
Phase 1 is single-filer (no cross-filer replication of the registry);
mounts that fail over to another filer will re-register on the next
heartbeat, so the registry self-heals within one TTL cycle.
Part of the peer-chunk-sharing design; no behavior change at runtime
until a later PR enables the flag on both filer and mount.
* filer: nil-safe peerRegistryEnable + registry hardening
Addresses review feedback on PR #9131.
* Fix: nil pointer deref in the mini cluster. FilerOptions instances
constructed outside weed/command/filer.go (e.g. miniFilerOptions in
mini.go) do not populate peerRegistryEnable, so dereferencing the
pointer panics at Filer startup. Use the same
`nil && deref` idiom already used for distributedLock / writebackCache.
* Hardening (gemini review): registry now enforces three invariants:
- empty peer_addr is silently rejected (no client-controlled sentinel
mass-inserts)
- TTL is capped at 1 hour so a runaway client cannot pin entries
- new-entry count is capped at 10000 to bound memory; renewals of
existing entries are always honored, so a full registry still
heartbeats its existing members correctly
Covered by new unit tests.
* filer: rename -peer.registry.enable flag to -mount.p2p
Per review feedback: the old name "peer.registry.enable" leaked
the implementation ("registry") into the CLI surface. "mount.p2p"
is shorter and describes what it actually controls — whether this
filer participates in mount-to-mount peer chunk sharing.
Flag renames (all three keep default=true, idle cost is near-zero):
-peer.registry.enable -> -mount.p2p (weed filer)
-filer.peer.registry.enable -> -filer.mount.p2p (weed mini, weed server)
Internal variable names (mountPeerRegistryEnable, MountPeerRegistry)
keep their longer form — they describe the component, not the knob.
* filer: MountList returns DataCenter + List uses RLock
Two review follow-ups on the mount peer registry:
* weed/server/filer_grpc_server_mount_peer.go: MountList was dropping
the DataCenter on the wire. The whole point of carrying DC separately
from Rack is letting the mount-side fetcher re-rank peers by the
two-level locality hierarchy (same-rack > same-DC > cross-DC); without
DC in the response every remote peer collapsed to "unknown locality."
* weed/filer/mount_peer_registry.go: List() was taking a write lock so
it could lazy-delete expired entries inline. But MountList is a
read-heavy RPC hit on every mount's 30 s refresh loop, and Sweep is
already wired as the sole reclamation path (same pattern as the
mount-side PeerDirectory). Switch List to RLock + filter, let Sweep
do the map mutation, so concurrent MountList callers don't serialize
on each other.
Test updated to reflect the new contract (List no longer mutates the
map; Sweep is what drops expired entries).
* mount: add peer chunk sharing options + advertise address resolver
First cut at the peer chunk sharing wiring on the mount side. No
functional behavior yet — this PR just introduces the option fields,
the -peer.* flags, and the helper that resolves a reachable
host:port from them. The server implementation arrives in PR #5
(gRPC service) and the fetcher in PR #7.
* ResolvePeerAdvertiseAddr: an explicit -peer.advertise wins; else we
use -peer.listen's bind host if specific; else util.DetectedHostAddress
combined with the port. This is what gets registered with the filer
and announced to peers, so wildcard binds no longer result in
unreachable identities like "[::]:18080".
* Option fields: PeerEnabled, PeerListen, PeerAdvertise, PeerRack.
One port handles both directory RPCs and streaming chunk fetches
(see PR #1 FetchChunk proto), so there is no second -peer.grpc.*
flag — the old HTTP byte-transfer path is gone.
* New flags on weed mount: -peer.enable, -peer.listen (default :18080),
-peer.advertise (default auto), -peer.rack.
* mount: register with filer and maintain HRW seed view
Adds the mount-side tier-1 client. On startup the mount calls
MountRegister with its advertise address (PR #3) and keeps both the
filer entry and the local seed view fresh via background tickers
(30 s register / 30 s list, 90 s filer TTL).
* peer_hrw.go: pure rendezvous-hashing helper picking a single owner
per fid via top-1 HRW. Adding or removing one seed moves only
~1/N fids.
* peer_registrar.go: heartbeat + list poller. Seeds() returns the
slice directly (no per-call copy) since listOnce atomically swaps;
background RPCs bind their context to Stop() so unmount doesn't
hang on a slow filer.
* WFS wiring uses ResolvePeerAdvertiseAddr from PR #3 for the
identity registered with the filer. No HTTP server, no second
port — one reachable address represents the mount.
* mount: broadcast MountRegister/MountList to every filer
Previously the registrar called through wfs.WithFilerClient, which only
reaches whichever filer the WFS filer-client session happens to be on.
That meant two mounts pointing at different filers would never see each
other: the filer mount registries are in-memory and per-filer (no
filer-to-filer sync), so each mount's MountList only returned peers
that had also registered through the same filer.
This commit makes the registrar multi-filer aware:
* NewPeerRegistrar now takes the full FilerAddresses slice and a
per-filer dial function. The old single-filer peerFilerClient
interface is gone.
* registerOnce fans a MountRegister RPC out to every filer in
parallel. Succeeds if at least one filer accepted — an unreachable
filer is tolerated, logged, and retried on the next heartbeat.
* listOnce polls every filer's MountList in parallel and merges the
responses by peer_addr, keeping the newest LastSeenNs on duplicates.
Mounts talking to different filers therefore converge once every
filer has been polled once.
The merged-list property is what lets a fleet of mounts spread across
multiple filers still form a single HRW seed view. Each filer only ever
sees the subset of mounts that heartbeat through it, but the registrar
reconstructs the union client-side.
New unit tests guard both properties:
- RegisterBroadcastsToAllFilers: one registerOnce hits all N filers.
- ListMergesAcrossFilers: mount-a on filer-1 and mount-b on filer-2
both appear in the merged seed set.
- ListMergeKeepsNewestLastSeen: the same mount reported by two
filers collapses to one entry with the freshest timestamp.
* proto: define MountRegister/MountList and MountPeer service
Adds the wire types for peer chunk sharing between weed mount clients:
* filer.proto: MountRegister / MountList RPCs so each mount can heartbeat
its peer-serve address into a filer-hosted registry, and refresh the
list of peers. Tiny payload; the filer stores only O(fleet_size) state.
* mount_peer.proto (new): ChunkAnnounce / ChunkLookup RPCs for the
mount-to-mount chunk directory. Each fid's directory entry lives on
an HRW-assigned mount; announces and lookups route to that mount.
No behavior yet — later PRs wire the RPCs into the filer and mount.
See design-weed-mount-peer-chunk-sharing.md for the full design.
* filer: add mount-server registry behind -peer.registry.enable
Implements tier 1 of the peer chunk sharing design: an in-memory registry
of live weed mount servers, keyed by peer address, refreshed by
MountRegister heartbeats and served by MountList.
* weed/filer/peer_registry.go: thread-safe map with TTL eviction; lazy
sweep on List plus a background sweeper goroutine for bounded memory.
* weed/server/filer_grpc_server_peer.go: MountRegister / MountList RPC
handlers. When -peer.registry.enable is false (the default), both RPCs
are silent no-ops so probing older filers is harmless.
* -peer.registry.enable flag on weed filer; FilerOption.PeerRegistryEnabled
wires it through.
Phase 1 is single-filer (no cross-filer replication of the registry);
mounts that fail over to another filer will re-register on the next
heartbeat, so the registry self-heals within one TTL cycle.
Part of the peer-chunk-sharing design; no behavior change at runtime
until a later PR enables the flag on both filer and mount.
* filer: nil-safe peerRegistryEnable + registry hardening
Addresses review feedback on PR #9131.
* Fix: nil pointer deref in the mini cluster. FilerOptions instances
constructed outside weed/command/filer.go (e.g. miniFilerOptions in
mini.go) do not populate peerRegistryEnable, so dereferencing the
pointer panics at Filer startup. Use the same
`nil && deref` idiom already used for distributedLock / writebackCache.
* Hardening (gemini review): registry now enforces three invariants:
- empty peer_addr is silently rejected (no client-controlled sentinel
mass-inserts)
- TTL is capped at 1 hour so a runaway client cannot pin entries
- new-entry count is capped at 10000 to bound memory; renewals of
existing entries are always honored, so a full registry still
heartbeats its existing members correctly
Covered by new unit tests.
* filer: rename -peer.registry.enable flag to -mount.p2p
Per review feedback: the old name "peer.registry.enable" leaked
the implementation ("registry") into the CLI surface. "mount.p2p"
is shorter and describes what it actually controls — whether this
filer participates in mount-to-mount peer chunk sharing.
Flag renames (all three keep default=true, idle cost is near-zero):
-peer.registry.enable -> -mount.p2p (weed filer)
-filer.peer.registry.enable -> -filer.mount.p2p (weed mini, weed server)
Internal variable names (mountPeerRegistryEnable, MountPeerRegistry)
keep their longer form — they describe the component, not the knob.
* filer: MountList returns DataCenter + List uses RLock
Two review follow-ups on the mount peer registry:
* weed/server/filer_grpc_server_mount_peer.go: MountList was dropping
the DataCenter on the wire. The whole point of carrying DC separately
from Rack is letting the mount-side fetcher re-rank peers by the
two-level locality hierarchy (same-rack > same-DC > cross-DC); without
DC in the response every remote peer collapsed to "unknown locality."
* weed/filer/mount_peer_registry.go: List() was taking a write lock so
it could lazy-delete expired entries inline. But MountList is a
read-heavy RPC hit on every mount's 30 s refresh loop, and Sweep is
already wired as the sole reclamation path (same pattern as the
mount-side PeerDirectory). Switch List to RLock + filter, let Sweep
do the map mutation, so concurrent MountList callers don't serialize
on each other.
Test updated to reflect the new contract (List no longer mutates the
map; Sweep is what drops expired entries).
* mount: add peer chunk sharing options + advertise address resolver
First cut at the peer chunk sharing wiring on the mount side. No
functional behavior yet — this PR just introduces the option fields,
the -peer.* flags, and the helper that resolves a reachable
host:port from them. The server implementation arrives in PR #5
(gRPC service) and the fetcher in PR #7.
* ResolvePeerAdvertiseAddr: an explicit -peer.advertise wins; else we
use -peer.listen's bind host if specific; else util.DetectedHostAddress
combined with the port. This is what gets registered with the filer
and announced to peers, so wildcard binds no longer result in
unreachable identities like "[::]:18080".
* Option fields: PeerEnabled, PeerListen, PeerAdvertise, PeerRack.
One port handles both directory RPCs and streaming chunk fetches
(see PR #1 FetchChunk proto), so there is no second -peer.grpc.*
flag — the old HTTP byte-transfer path is gone.
* New flags on weed mount: -peer.enable, -peer.listen (default :18080),
-peer.advertise (default auto), -peer.rack.
* proto: define MountRegister/MountList and MountPeer service
Adds the wire types for peer chunk sharing between weed mount clients:
* filer.proto: MountRegister / MountList RPCs so each mount can heartbeat
its peer-serve address into a filer-hosted registry, and refresh the
list of peers. Tiny payload; the filer stores only O(fleet_size) state.
* mount_peer.proto (new): ChunkAnnounce / ChunkLookup RPCs for the
mount-to-mount chunk directory. Each fid's directory entry lives on
an HRW-assigned mount; announces and lookups route to that mount.
No behavior yet — later PRs wire the RPCs into the filer and mount.
See design-weed-mount-peer-chunk-sharing.md for the full design.
* filer: add mount-server registry behind -peer.registry.enable
Implements tier 1 of the peer chunk sharing design: an in-memory registry
of live weed mount servers, keyed by peer address, refreshed by
MountRegister heartbeats and served by MountList.
* weed/filer/peer_registry.go: thread-safe map with TTL eviction; lazy
sweep on List plus a background sweeper goroutine for bounded memory.
* weed/server/filer_grpc_server_peer.go: MountRegister / MountList RPC
handlers. When -peer.registry.enable is false (the default), both RPCs
are silent no-ops so probing older filers is harmless.
* -peer.registry.enable flag on weed filer; FilerOption.PeerRegistryEnabled
wires it through.
Phase 1 is single-filer (no cross-filer replication of the registry);
mounts that fail over to another filer will re-register on the next
heartbeat, so the registry self-heals within one TTL cycle.
Part of the peer-chunk-sharing design; no behavior change at runtime
until a later PR enables the flag on both filer and mount.
* filer: nil-safe peerRegistryEnable + registry hardening
Addresses review feedback on PR #9131.
* Fix: nil pointer deref in the mini cluster. FilerOptions instances
constructed outside weed/command/filer.go (e.g. miniFilerOptions in
mini.go) do not populate peerRegistryEnable, so dereferencing the
pointer panics at Filer startup. Use the same
`nil && deref` idiom already used for distributedLock / writebackCache.
* Hardening (gemini review): registry now enforces three invariants:
- empty peer_addr is silently rejected (no client-controlled sentinel
mass-inserts)
- TTL is capped at 1 hour so a runaway client cannot pin entries
- new-entry count is capped at 10000 to bound memory; renewals of
existing entries are always honored, so a full registry still
heartbeats its existing members correctly
Covered by new unit tests.
* filer: rename -peer.registry.enable flag to -mount.p2p
Per review feedback: the old name "peer.registry.enable" leaked
the implementation ("registry") into the CLI surface. "mount.p2p"
is shorter and describes what it actually controls — whether this
filer participates in mount-to-mount peer chunk sharing.
Flag renames (all three keep default=true, idle cost is near-zero):
-peer.registry.enable -> -mount.p2p (weed filer)
-filer.peer.registry.enable -> -filer.mount.p2p (weed mini, weed server)
Internal variable names (mountPeerRegistryEnable, MountPeerRegistry)
keep their longer form — they describe the component, not the knob.
* filer: MountList returns DataCenter + List uses RLock
Two review follow-ups on the mount peer registry:
* weed/server/filer_grpc_server_mount_peer.go: MountList was dropping
the DataCenter on the wire. The whole point of carrying DC separately
from Rack is letting the mount-side fetcher re-rank peers by the
two-level locality hierarchy (same-rack > same-DC > cross-DC); without
DC in the response every remote peer collapsed to "unknown locality."
* weed/filer/mount_peer_registry.go: List() was taking a write lock so
it could lazy-delete expired entries inline. But MountList is a
read-heavy RPC hit on every mount's 30 s refresh loop, and Sweep is
already wired as the sole reclamation path (same pattern as the
mount-side PeerDirectory). Switch List to RLock + filter, let Sweep
do the map mutation, so concurrent MountList callers don't serialize
on each other.
Test updated to reflect the new contract (List no longer mutates the
map; Sweep is what drops expired entries).
proto: define MountRegister/MountList and MountPeer service
Adds the wire types for peer chunk sharing between weed mount clients:
* filer.proto: MountRegister / MountList RPCs so each mount can heartbeat
its peer-serve address into a filer-hosted registry, and refresh the
list of peers. Tiny payload; the filer stores only O(fleet_size) state.
* mount_peer.proto (new): ChunkAnnounce / ChunkLookup RPCs for the
mount-to-mount chunk directory. Each fid's directory entry lives on
an HRW-assigned mount; announces and lookups route to that mount.
No behavior yet — later PRs wire the RPCs into the filer and mount.
See design-weed-mount-peer-chunk-sharing.md for the full design.
* fix(test): reduce kafka gateway and consumer group flakes
* fix(kafka): make broker health-check backoff respect context
Replace time.Sleep in the retry loop with a select on bc.ctx.Done() and
time.After so the backoff is interruptible during shutdown, per review
feedback on PR #9129.
* fix(kafka): guard broker HealthCheck against nil client
Return the same "broker client not connected" error used by the other
exported BrokerClient methods instead of panicking on a partially
initialized client, per CodeRabbit review feedback on PR #9129.
ceph/s3-tests pins lxml without an upper bound. When pip picks a release
whose prebuilt wheel isn't published for Python 3.9 on the runner, it
falls back to sdist and fails without libxml2-dev / libxslt1-dev.
* fix(mount): close inodeLocks cleanup race that allowed two flock holders
PosixLockTable.getOrCreateInodeLocks released plt.mu before the caller
acquired il.mu. A concurrent maybeCleanupInode could delete the map
entry in that window; the first caller would then insert its lock into
the orphaned inodeLocks while a later caller created a fresh entry in
the map, so findConflict never observed the orphaned lock and two
owners could simultaneously believe they held the same exclusive flock.
This matches the flaky CI failure seen in
TestPosixFileLocking/ConcurrentLockContention:
Error: Should be empty, but was [worker N: flock overlap detected with 2 holders]
Mark removed inodeLocks as dead under plt.mu+il.mu, and have SetLk /
SetLkw recheck the flag after locking il.mu, refetching the live entry
from the map when orphaned. Also delete the map entry only if it still
points to this il, so a racing recreate is not clobbered.
Adds TestConcurrentFlockChurnPreservesMutualExclusion: 16 goroutines x
500 flock/unflock iterations on one inode. Reliably reports 500+
overlaps per run before the fix; clean across 100 race-enabled runs
after.
* fix(mount): extend dead-flag contract to GetLk and self-heal primitives
Address review feedback on the initial cleanup-race fix:
1. GetLk had the same stale-pointer bug as SetLk. A caller could grab
an inodeLocks pointer, have cleanup orphan it and a replacement il
receive a conflicting lock, then answer F_UNLCK off the empty dead
pointer. Add the same dead recheck + refetch loop.
2. getOrCreateInodeLocks and getInodeLocks now treat a dead map entry
as defective: the former replaces it with a fresh inodeLocks, the
latter drops it and returns nil. Production cannot reach that state
(maybeCleanupInode atomically deletes under plt.mu when it sets
dead), but the hardening guarantees the SetLk / SetLkw / GetLk
retry loops always make progress even if a future refactor reorders
those operations, and it lets the white-box tests set up a stale
dead entry without spinning.
3. Strengthen the regression suite:
- TestSetLkRetriesPastDeadInodeLocks: deterministic white-box test
that installs a dead il in the map and asserts SetLk routes the
new lock into a fresh il (not the orphan), that GetLk reports the
resulting conflict, and that a different-owner acquire is rejected
with EAGAIN.
- TestGetInodeLocksEvictsDeadEntry: verifies both map-read primitives
drop or replace dead entries.
- TestConcurrentFlockChurnPreservesMutualExclusion: replace the
timing-fragile Add(1)-and-check counter with a Swap+CAS detector.
Each worker claims a slot after SetLk OK and releases it before
UN, flagging both an observed predecessor and a lost CAS on
release. Against a reverted fix the detector fires 1000+ times per
run; with the fix clean across 100 race-enabled iterations.
* test(mount): fail fast on unexpected SetLk statuses in churn loop
The stress test blindly spun on any non-OK SetLk status and discarded
the unlock return. If SetLk ever returns something other than OK or
EAGAIN (e.g. after a future refactor introduces a new error), the
acquire loop would spin forever and an unlock failure would be
silently swallowed.
Capture the acquire status, retry only on the expected EAGAIN, and
assert unlock returns OK. Use t.Errorf + return (not t.Fatalf) because
the checks run on worker goroutines where FailNow is unsafe. The
Swap+CAS overlap detector is unchanged.
* feat(filer.backup): -initialSnapshot seeds destination from live tree
Replaying the metadata event log on a fresh sync only leaves files that
still exist on the source at replay time: any entry that was created and
later deleted is replayed as a create/delete pair and never materializes
on the destination. Users who wipe the destination and re-run
filer.backup therefore see "only new files" instead of a full backup,
even when -timeAgo=876000h is passed and the subscription genuinely
starts from epoch (ref discussion #8672).
Add a -initialSnapshot opt-in flag: when set on a fresh sync (no prior
checkpoint, -timeAgo unset), walk the live filer tree under -filerPath
via TraverseBfs and seed the destination through sink.CreateEntry, then
persist the walk-start timestamp as the checkpoint and subscribe from
there. Capturing the timestamp before the walk lets the subscription
catch any create/update/delete racing with the walk — sink CreateEntry
is idempotent across the builtin sinks so replay is safe.
Honors existing -filerExcludePaths / -filerExcludeFileNames /
-filerExcludePathPatterns filters and skips /topics/.system/log the
same way the subscription path does.
Also log "starting from <t> (no prior checkpoint)" instead of a
misleading "resuming from 1970-01-01" when the KV has no stored offset.
* fix(filer.backup): guard initialSnapshot counters under TraverseBfs workers
TraverseBfs fans the callback out across 5 worker goroutines, so the
entryCount / byteCount updates and the 5-second progress-log gate in
runInitialSnapshot were racing. Switch the counters to atomic.Int64 and
protect the lastLog check/update with a short-scoped mutex so the heavy
sink.CreateEntry call stays outside the critical section.
Flagged by gemini-code-assist on #9126; verified with go test -race.
* fix(filer.backup): harden initialSnapshot against transient errors and path edge cases
Three review items from CodeRabbit on #9126:
1. getOffset errors no longer leave isFreshSync=true. Before, a transient
KV read failure would cause runFilerBackup's retry loop to redo the
full -initialSnapshot walk on every retry. Treat any offset-read
error as "not fresh" so the snapshot only runs when we've verified
there really is no prior checkpoint.
2. initialSnapshotTargetKey now normalizes sourcePath to a trailing-
slash base before stripping the prefix, so edge cases where
sourceKey equals sourcePath (trailing-slash mismatch or root-entry
emission) no longer index past the end. Unit tests cover both
forms.
3. Documented the TraverseBfs-enumerates-excluded-subtrees performance
characteristic on runInitialSnapshot, since pruning requires a
separate change to TraverseBfs itself.
* fix(filer.backup): retry setOffset after initialSnapshot to avoid full re-walks
If the snapshot walk finishes but the subsequent setOffset fails, the
retry loop in runFilerBackup will re-enter doFilerBackup with an empty
checkpoint and run the full BFS again — on a multi-million-entry tree
that's hours of wasted work over a 100-byte KV write. Retry the write a
handful of times with exponential backoff before giving up, and log
loudly at the final failure (with snapshotTsNs + sinkId) so operators
recognize the symptom instead of guessing at mysterious repeated walks.
Nitpick raised by CodeRabbit on #9126.
* fix(filer.backup): initialSnapshot ignore404, skew margin, exclude dir-entry itself
Three review items from CodeRabbit on #9126:
1. ignore404Error now threads into runInitialSnapshot. If a file is listed
by TraverseBfs and then deleted before CreateEntry reads its chunks,
the follow path already ignores 404s — the snapshot path was aborting
and triggering a full re-walk. Treat an ignorable 404 as "skip this
entry, continue."
2. snapshotTsNs now uses `time.Now() - 1min` instead of `time.Now()`.
Metadata events are stamped server-side, so a fast backup-host clock
could skip events that fire during or right after the walk. Matches
the 1-minute margin meta_aggregator.go applies on initial peer
traversal; duplicate replay is harmless because CreateEntry is
idempotent.
3. Exclude checks now run against the entry's own full path, not just
its parent. A walked directory whose full path matches SystemLogDir
or -filerExcludePaths was being seeded to the destination; only its
descendants were being skipped. Verified with a manual repro where
-filerExcludePaths=/data/skipdir now keeps the skipdir entry itself
off the destination.
* refactor(filer): share destKey helper between buildKey and initialSnapshot
Extract destKey(dataSink, targetPath, sourcePath, sourceKey, mTime) from
buildKey in filer_sync.go. Both the event-log path (buildKey) and the
initialSnapshot walk (initialSnapshotTargetKey) now go through the same
helper, so a walk-seeded file and an event-replayed file always resolve
to the same destination key.
As a bonus, buildKey picks up the defensive trailing-slash normalization
that initialSnapshotTargetKey introduced — no more index-past-end risk
when sourceKey happens to equal sourcePath. Also tightens the mTime
lookup to guard against nil Attributes (caught by an existing test
against buildKey when I first moved the lookup out of the incremental
branch).
* fix(shell): fs.mergeVolumes now rewrites manifest chunks for large files
Previously fs.mergeVolumes skipped any chunk whose IsChunkManifest flag was
true, printing "Change volume id for large file is not implemented yet" and
continuing. Because the BFS traversal only looks at top-level
entry.Chunks, sub-chunks referenced inside a manifest were never
considered either. For any file stored as a chunk manifest (large files
go this path), chunks in the source volume stayed put, leaving behind a
few MB of live data that vacuum and volume.deleteEmpty couldn't clean
up.
This change resolves each manifest chunk recursively, moves any
sub-chunk whose volume id is in the merge plan via the existing
moveChunk path, and re-serializes the manifest. If the manifest chunk
itself lives in a source volume, or any sub-chunk moved, the new
manifest blob is uploaded to a freshly assigned file id (the old
needle becomes orphaned and is reclaimed by vacuum like any other
moved chunk).
Fixes#9116.
* address review: batch UpdateEntry, fix dry-run, defer restore, avoid source volumes
- Call UpdateEntry once per entry after the chunk loop instead of once per
moved chunk (gemini nit).
- In dry-run mode, mark anySubChanged when a sub-chunk in the plan is
encountered and return changed=true after printing "rewrite manifest",
so nested manifests also surface their would-rewrites (gemini nit).
- Defer filer_pb.AfterEntryDeserialization so the manifest chunk list is
restored even when proto.Marshal fails (coderabbit nit).
- Reject AssignVolume results whose file id lands on a volume that is a
source in the merge plan, and retry — otherwise the replacement
manifest could be written to the volume being emptied (coderabbit).
* feat(shell): add fs.distributeChunks command for even chunk distribution
Add a new weed shell command that redistributes a file's chunks evenly
across volume server nodes.
Supports three distribution modes via -mode flag:
- primary: balance chunk ownership across nodes (default)
- replica: balance both ownership and replica copies
- round-robin: assign chunks by offset order for sequential read
optimization (chunk[0]->A, chunk[1]->B, chunk[2]->C, ...)
Additional options:
- -nodes=N to target specific number of nodes
- -apply to execute (dry-run by default)
Usage:
fs.distributeChunks -path=/buckets/file.dat
fs.distributeChunks -path=/buckets/file.dat -mode=round-robin -apply
fs.distributeChunks -path=/buckets/file.dat -mode=replica -apply
fs.distributeChunks -path=/buckets/file.dat -nodes=5 -apply
* fix(shell): improve fs.distributeChunks robustness and code quality
- Propagate flag parse errors instead of swallowing them (return err)
- Handle nil chunk.Fid by falling back to legacy FileId string parsing
- Simplify node membership check using slices.Contains
* fix(shell): fix dead round-robin print loop in fs.distributeChunks
The loop was computing targetNode with sc.index%totalNodes (original
chunk index) instead of the sequential position, and discarding it via
_ = targetNode without printing anything. Replace with a correct loop
using pos%totalNodes and actually print the first 12 node assignments.
* fix(shell): compute replication/collection per-chunk in fs.distributeChunks
Previously replication and collection were derived once from chunks[0]
and reused for all moves, causing wrong volume placement for chunks
belonging to different volumes or collections. Now each chunk looks up
its own volumeInfoMap entry immediately before calling operation.Assign.
* fix(shell): prefer assignResult.Auth JWT over local signing key in fs.distributeChunks
When the master returns an Auth token in the Assign response, use it
directly for the upload instead of generating a new JWT from the local
viper signing key. Fall back to local key generation only when Auth is
empty, matching the pattern used by other upload paths.
* fix(shell): add timeout and error handling to delete requests in fs.distributeChunks
The delete loop was ignoring http.NewRequest errors and had no timeout,
risking a nil-request panic or indefinite block. Replace with
http.NewRequestWithContext and a 30s timeout, handle request creation
errors by incrementing deleteFailCount, and cancel the context
immediately after Do returns.
* feat(shell): parallelize chunk moves in fs.distributeChunks using ErrorWaitGroup
Sequential chunk moves are a bottleneck for large LLM model files with
hundreds or thousands of chunks. Use ErrorWaitGroup with
DefaultMaxParallelization (10) to run download/assign/upload concurrently.
Guard movedRecords appends, chunk.Fid updates, and writer output with a
mutex. Individual chunk failures are non-fatal and logged inline; only
successfully moved chunks are included in the metadata update.
* fix(shell): try all replica URLs on download in fs.distributeChunks
Previously only the first volume server URL was attempted, causing chunk
moves to fail if that replica was unreachable. Now iterates through all
URLs returned by LookupVolumeServerUrl and stops at the first success.
* refactor(shell): apply extract method pattern to fs.distributeChunks
Do() was a single ~615-line function. Break it into focused helpers:
- lookupFileEntry: filer entry lookup
- validateChunks: chunk manifest guard
- collectVolumeTopology: master topology query + ownership mapping
- buildDistributionCounts: chunk→node mapping and owner/copy tallies
- selectActiveNodes: target node selection
- printCurrentDistribution: per-node distribution table
- planDistribution: mode-switch planning (primary/replica/round-robin)
- printRedistributionPlan: before/after plan table
- relevantNodes: active-or-occupied node filter
Do() is now ~100 lines of orchestration; each helper has a single
clear responsibility.
* test(shell): add unit tests for fs.distributeChunks algorithms
Cover all three distribution modes and supporting helpers:
- shortName, relevantNodes
- computeOwnerTarget (even/uneven split, inactive node drain)
- buildDistributionCounts (normal + nil Fid fallback)
- selectActiveNodes (all nodes / limited count)
- planOwnerMoves (imbalanced → balanced, already balanced)
- planDistribution primary (chunks balanced, no-op when even)
- planDistribution round-robin (offset ordering, correct assignment)
- planDistribution replica (owner + copy balancing)
- printRedistributionPlan (output format)
* fix(shell): add 5-minute timeout to chunk downloads in fs.distributeChunks
Download requests had no per-request timeout, unlike delete operations
which already use 30s. Replace readUrl() calls with inline
http.NewRequestWithContext + context.WithTimeout(5m) so a hung volume
server cannot block a goroutine indefinitely during redistribution.
* fix(shell): remove redundant deleteOldChunks in fs.distributeChunks
filer.UpdateEntry already calls deleteChunksIfNotNew internally, which
computes the diff between old and new entry chunks and deletes the ones
no longer referenced. Our explicit deleteOldChunks was racing with this
filer-side cleanup, causing spurious 404 warnings on ~75% of deletes.
Remove deleteOldChunks, movedChunkRecord type, and reduce
executeChunkMoves return type to (int, error) for the moved count.
* fix(shell): handle nil chunk.Fid via chunkVolumeId helper in fs.distributeChunks
chunk.Fid.GetVolumeId() silently returns 0 for legacy chunks stored with
a FileId string instead of a Fid struct, causing them to be skipped in
the replica balancing loop and looked up incorrectly in volumeInfoMap.
Introduce chunkVolumeId() that uses Fid when present and falls back to
parsing the legacy FileId string, matching the logic in
buildDistributionCounts. Apply it in the replica-mode copies loop and
in executeChunkMoves' replication/collection lookup.
* fix(shell): use already-parsed oldFid for volumeInfoMap lookup in fs.distributeChunks
chunkVolumeId(chunk) was being called to look up replication/collection
after oldFid had already been parsed and validated. Use oldFid.VolumeId
directly to avoid redundant parsing and guarantee the correct volume ID
regardless of whether chunk.Fid is nil.
* fix(shell): improve correctness and robustness in fs.distributeChunks
- Buffer download body before upload so dlCtx timeout only covers the
GET request; upload runs with context.Background() via bytes.NewReader
- Replace 'before, after := strings.Cut(...)' + '_ = before' with '_'
as the first return value directly
- Clone copiesCount before replica planner mutates it, keeping the
caller's map immutable
- Add nil-entry guard after filer LookupEntry to prevent panic on
unexpected nil response
* feat(shell): support chunk manifests in fs.distributeChunks
Large files stored as chunk manifests were previously rejected. Resolve
manifests up front via filer.ResolveChunkManifest, redistribute the
underlying data chunks, then re-pack through filer.MaybeManifestize
before UpdateEntry. The filer's MinusChunks resolves manifests on both
sides of the diff, so old manifest and inner data chunks are GC'd
automatically.
* fix(shell): match master's SaveDataAsChunkFunctionType 5-param signature
Master added expectedDataSize uint64; ignore it in shell-side saveAsChunk.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(s3api): self-heal stale .versions latest-version pointer on read
When the `.versions` directory metadata points at a version file that has
gone missing (e.g. a crash between deleting the latest version and
rewriting the pointer, or a concurrent delete racing with a read),
`getLatestObjectVersion` bailed with a hard error that required manual
repair. Tagging, ACL, retention, copy-source, and HEAD/GET all surfaced
NoSuchKey even though other versions remained on disk.
On `ErrNotFound`/`codes.NotFound` from the pointed-at version lookup,
rescan the `.versions` directory, pick the newest remaining non-delete-
marker entry, persist the repaired pointer (best-effort), and return
that entry. If only delete markers (or nothing) remain, the caller still
sees an error and the object correctly appears absent.
Extracted the selection logic into a pure `selectLatestContentVersion`
helper so `updateLatestVersionAfterDeletion` and the new self-heal path
share a single implementation. A warning is logged whenever the heal
kicks in so stale-pointer incidents remain visible in operator logs.
* fix(s3api): self-heal must promote newest version even if delete marker
Review feedback (gemini-code-assist): the self-heal path used
`selectLatestContentVersion`, which skips delete markers. That had two
bugs:
1. If the chronologically newest entry was a delete marker, an older
content version would be promoted, effectively "undeleting" an
object that was actually deleted.
2. If only delete markers remained, heal returned an error and the
caller surfaced a hard 500 instead of the correct 404-with-
x-amz-delete-marker response.
Add `selectLatestVersion` that picks the newest entry regardless of
type (content or delete marker) and use it in
`healStaleLatestVersionPointer`. The promoted entry flows back through
`doGetLatestObjectVersion` unchanged; downstream handlers already
detect `ExtDeleteMarkerKey` on the returned entry and render
NoSuchKey + `x-amz-delete-marker: true` (see
`s3api_object_handlers.go:722-728`).
Kept `selectLatestContentVersion` in place for
`updateLatestVersionAfterDeletion`, which deliberately limits the
pointer to a live content version in the post-deletion flow — changing
that is out of scope for this fix.
Added four tests for the new selector:
- promotes newest delete marker over older content (the reviewer case)
- picks content when content is newest
- promotes newest delete marker when only delete markers remain
- returns nil latestEntry on empty/untagged input
* fix(s3api): paginate .versions scan in self-heal path
Review feedback (gemini-code-assist, coderabbitai): the self-heal rescan
did a single-shot list(..., 1000). For objects whose version ids use the
old raw-timestamp format, filer ordering is lexicographic-ascending =
oldest-first. If the .versions directory held more than one page of
entries, the first page contained only the oldest, and the heal would
promote an older version as the new "latest" — silently surfacing stale
data on subsequent reads.
Paginate through the whole directory with `filer.PaginationSize`,
running `selectLatestVersion` per page and keeping a single running
best candidate across pages (via `compareVersionIds`). This mirrors the
pagination pattern already used by `getObjectVersionList` in the same
file and closes the window for old-format stacks larger than one page.
New-format (inverted-timestamp) ids were not affected because their
lexicographic order matches newest-first, but paginating is still the
right fix.
Also updated the function doc to reflect that self-heal now promotes
the newest entry regardless of type (content version or delete marker).
* fix(s3api): don't resurrect deleted objects; wrap ErrNotFound sentinel
Two review findings addressed:
1. `updateLatestVersionAfterDeletion` was using `selectLatestContentVersion`
which skipped delete markers. Scenario: PUT v1, DELETE (dm1 written,
pointer->dm1), PUT v3 (pointer->v3), user explicitly deletes v3 by
versionId. Remaining files: v1, dm1. S3 semantics say the current
version is the chronologically newest = dm1 (object appears deleted).
Old code would promote v1, "undeleting" the object silently.
Switched to `selectLatestVersion`, which picks the newest entry
regardless of type. Also paginated the scan the same way the self-heal
path does — otherwise a single-page `list(1000)` still mis-selects the
latest for old-format version id stacks that exceed one page.
Removed the `hasDeleteMarkers || !isLast` branch: with `selectLatestVersion`
any delete marker participates in the comparison and shows up as the
latest when it is newest, so the "keep the directory on ambiguity"
guard becomes unreachable. Full pagination also makes the `!isLast`
guard unnecessary — we either see every entry or surface a list error.
2. `healStaleLatestVersionPointer`'s "no remaining version" error was a
plain `fmt.Errorf`, so callers could not distinguish genuine
object-absence from a scan failure via `errors.Is`. Wrap it with
`filer_pb.ErrNotFound` so the sentinel flows through the chain (the
outer wrap already uses `%w`).
No test additions needed — `TestSelectLatestVersion_PromotesNewestDeleteMarker`
already asserts the "newer delete marker beats older content" invariant
that drives both fixes.
* refactor(s3api): drop unused selectLatestContentVersion after review
Review feedback flagged a stale comment claiming
selectLatestContentVersion "mirrors" the post-deletion semantics of
updateLatestVersionAfterDeletion. That claim became false when the
post-deletion path switched to selectLatestVersion in the previous
commit. Verified no production callers remain — only the helper's own
tests referenced it — so the cleaner fix is to delete the dead code
rather than rewrite the comment to explain "this exists but isn't
used."
- Removed selectLatestContentVersion.
- Removed the four TestSelectLatestContentVersion_* tests.
- Preserved a renamed TestSelectLatestVersion_MixedFormats so the
mixed-format comparator coverage still runs against the active
selector.
* fix(s3): reject unknown POST policy conditions and extra x-amz form fields
CheckPostPolicy previously accepted policy conditions with unknown $keys
(e.g. "$foo") as satisfied, and only rejected stray X-Amz-Meta-* form
fields. Reject unknown condition keys outright, and extend the extra-
input-fields check to all X-Amz-* form fields except the reserved
auth/signing headers. Matches AWS S3 POST Object behavior.
* refactor(s3): drop redundant $x-amz-meta- prefix check in CheckPostPolicy
The $x-amz- prefix already subsumes $x-amz-meta-, so the explicit
$x-amz-meta- check adds no coverage. Simplify the else-if condition.
Addresses gemini-code-assist review on PR #9124.
* style(s3): align unknown-key policy error with [op, key, value] trailer
Reformat the unknown-condition-key error in CheckPostPolicy to include
the same "[op, key, value]" trailer used by the other condition-failed
messages. The value slot is empty because no comparison occurs for an
unknown key. The descriptive "unknown condition key" suffix is kept so
operators can still tell this failure from a mismatched value.
* fix(s3): honor starts-with prefix-stem POST policies when checking extras
AWS POST policies use ["starts-with","$x-amz-meta-",""] to allow any
X-Amz-Meta-* form field. The previous exact-match policyXAmzKeys would
flag every X-Amz-Meta-Foo as an "Extra input fields" failure because
only the stem X-Amz-Meta- was stored. Track starts-with conditions whose
key ends in "-" with an empty value as prefix stems, and accept any
X-Amz-* form field matching one of those stems.
* fix(s3): validate value prefix for starts-with POST policy stems
Drop the policy.Value == "" gate when detecting prefix-stem conditions
so that ["starts-with","$x-amz-meta-","pfx-"] is recognized as a prefix
rule. Track the required value prefix alongside the name prefix, enforce
it against every matching form field in the extras loop, and skip the
prefix-stem condition in the main iteration (it has no single form
field to evaluate). Also include policy.Value in the unknown-condition
error trailer for clearer debugging. Addresses gemini-code-assist
review on PR #9124.
* fix(s3): check every matching POST policy rule, not just the first
The extras loop exited early on exact-key match and broke on the first
matching prefix stem. Per AWS, a form field must satisfy every policy
condition that applies to it, so an exact-match field must still honor
any overlapping starts-with stem's value prefix, and multiple stems on
the same field must all hold. Drop both early exits: start matched from
the exact-key lookup, iterate all prefix stems, and fail on the first
value-prefix violation. Addresses gemini-code-assist review on PR
#9124.
* fix(s3): propagate validated POST form fields to upload headers
POST Object form fields like acl, Content-Encoding, x-amz-storage-class,
x-amz-tagging, and x-amz-server-side-encryption were validated against
the POST policy but never forwarded to the underlying PUT, so the
validated values had no effect. Forward all non-reserved x-amz-* fields
plus acl (as x-amz-acl) and Content-Encoding. Reserved POST policy
mechanism fields (Policy, Signature, Key, etc.) are still excluded.
* fix(s3): also forward Content-Language from POST form to upload headers
AWS S3 POST Object supports Content-Language; add it to the set of
content headers forwarded by applyPostPolicyFormHeaders alongside
Cache-Control, Expires, Content-Disposition, and Content-Encoding.
Addresses gemini-code-assist review on PR #9123.
* refactor(s3): only look up form value in branches that use it
applyPostPolicyFormHeaders previously called formValues.Get(k) for every
form field, including fields that fall through the switch (non-reserved
fields that are neither Acl, a forwarded content header, nor X-Amz-*).
Move the lookup inside the switch cases that actually use it.
* fix(s3): return 403 on POST policy violation instead of 307 redirect
CheckPostPolicy failures previously responded with HTTP 307 Temporary
Redirect to the request URL, which causes clients to re-POST and
obscures the failure. Return 403 AccessDenied so the client surfaces
the error.
* test(s3): exercise PostPolicyBucketHandler end-to-end for 403 mapping
Replace the shallow ErrAccessDenied tautology test with one that builds
a signed POST multipart request whose policy conditions cannot be
satisfied, calls PostPolicyBucketHandler directly, and asserts HTTP 403
with no Location redirect header. Addresses gemini-code-assist review on
PR #9122.
* fix(s3): surface POST policy failure reason in AccessDenied response
Add s3err.WriteErrorResponseWithMessage so a caller can keep the
standard error code mapping while providing a specific Message. Use it
from PostPolicyBucketHandler so the XML body carries the CheckPostPolicy
error (e.g. which condition failed or that the policy expired) rather
than the generic "Access Denied." description. Addresses gemini-code-
assist review on PR #9122.
* refactor(s3err): delegate WriteErrorResponse to WriteErrorResponseWithMessage
The two helpers shared every line except the Message override. Fold
WriteErrorResponse into a one-line delegation that passes an empty
message, so the request-id/mux/apiError logic lives in exactly one
place. Addresses gemini-code-assist review on PR #9122.
freePort allocated in [20000, 55535], which overlaps the Linux ephemeral
range (32768-60999). The kernel could reuse the chosen port for an
outbound connection between the test closing its listener and weed mini
re-checking availability, causing:
Port allocation failed: port N for Filer (specified by flag
filer.port) is not available on 0.0.0.0 and cannot be used
Narrow the range to [20000, 32000] to stay below the ephemeral floor,
and pass -ip.bind=127.0.0.1 so mini's pre-check runs on the same IP the
test actually reserved the port on.
* fix(s3): strip client-supplied X-SeaweedFS-Principal/Session-Token in AuthSignatureOnly
AuthSignatureOnly is the only auth gate in front of S3Tables routes
(incl. CreateTableBucket) and UnifiedPostHandler, but unlike
authenticateRequestInternal it did not clear the internal IAM
trust headers before running signature verification. S3Tables
authorizeIAMAction reads X-SeaweedFS-Principal directly from the
request and prefers it over the authenticated identity's PrincipalArn,
so a signed low-privilege caller could append that header after signing
(unsigned header, SigV4 still verifies) and have IAM policy evaluated
against a spoofed principal, bypassing authorization.
Clear both X-SeaweedFS-Principal and X-SeaweedFS-Session-Token at the
top of AuthSignatureOnly, mirroring the existing guard in
authenticateRequestInternal. Add a regression test covering the
header-injection path.
* refactor(s3): route AuthSignatureOnly through authenticateRequestInternal
Addresses review feedback: both entry points were independently
maintaining the internal-IAM-header stripping and the auth-type dispatch
switch. Collapse AuthSignatureOnly into a thin wrapper around
authenticateRequestInternal so the security-critical header scrub and
the signature-verify switch live in one place. Post-auth behavior
unique to AuthSignatureOnly (AmzAccountId header) stays inline.
No functional change beyond two harmless telemetry tweaks that now
match authenticateRequestInternal: the per-branch glog verbosity shifts
from V(3) to V(4), and the anonymous-found path now sets AmzAuthType.
* refactor(s3): centralize X-SeaweedFS-Principal/Session-Token header names
Introduce SeaweedFSPrincipalHeader and SeaweedFSSessionTokenHeader in
weed/s3api/s3_constants so the trust-header literals are defined once and
referenced consistently by the auth scrub, JWT auth path, bucket policy
principal resolution, IAM authorization, and S3Tables IAM evaluation.
Replace every remaining usage in weed/s3api and weed/s3api/s3tables.
This removes the drift risk the reviewer called out: adding another call
site with a typo can no longer silently bypass the scrub.
Pure rename, no behavior change. No-op integration-test helper in
test/s3/iam/s3_iam_framework.go left untouched (separate module, and the
server now strips the client-supplied value regardless).
A presigned URL holder could attach arbitrary x-amz-* headers to a PUT
request (e.g. x-amz-tagging, x-amz-acl, x-amz-storage-class,
x-amz-server-side-encryption*, x-amz-object-lock-*, x-amz-meta-*,
x-amz-website-redirect-location, x-amz-grant-*). Because only the
headers declared in SignedHeaders participate in signature
verification, the added headers bypass authentication; the PUT handler
then persists them into the object's Extended metadata.
Match the AWS SigV4 rule: every x-amz-* header present in the request
must appear in SignedHeaders. Exempt x-amz-content-sha256 (already
tamper-protected via the canonical request's payload-hash line) and,
for presigned URLs, the SigV4 protocol parameters that live in the
query string (X-Amz-Algorithm/Credential/Date/Expires/SignedHeaders/
Signature) in case they are duplicated as headers.
Applies to both header-based and presigned SigV4; non-amz headers are
unaffected.
* ci(kafka-loadtest): retry apt-get to survive Ubuntu mirror flakes
The Kafka Quick Test workflow's Docker build of Dockerfile.loadtest
keeps hitting "Connection failed [IP: ...]" on archive.ubuntu.com /
security.ubuntu.com mid-build, e.g.:
failed to solve: process "/bin/sh -c apt-get update && \
apt-get install -y ca-certificates curl jq bash netcat ..."
did not complete successfully: exit code: 100
Same class of failure PR #9106 fixed for pjdfstest. Apply the same
two retry knobs to Dockerfile.loadtest and Dockerfile.seektest so a
transient mirror flake retries five times with a 30s timeout instead
of failing the whole workflow.
(The fuller pjdfstest fix also restructured the build to use
docker/build-push-action with type=gha cache. Doing that for
kafka-client-loadtest would mean rewriting the make/docker-compose
build path; defer until the apt-retry alone proves insufficient.)
* ci(kafka-loadtest): also drop recommends/suggests + apt-get clean
Address PR review (gemini-code-assist): fully align with PR #9106's
pattern by adding --no-install-recommends / --no-install-suggests so
the runtime images stay small and don't pull in extra packages, plus
apt-get clean before rm -rf /var/lib/apt/lists/* in Dockerfile.seektest.
* ci(kafka-loadtest): use alpine / maven base images instead of apt
The previous rounds of apt-retry / apt-clean knobs aren't enough:
the Ubuntu mirror is persistently unreachable from the GitHub runner
for minutes at a time, which blows past the 5-retry / 30-second
Acquire configuration and still kills the build (see run 24551809614).
Switch both runtime images so no apt fetch is needed at all:
- Dockerfile.loadtest now runs on alpine:3.20. All runtime deps
(ca-certificates, curl, jq, bash, netcat-openbsd) are in the
Alpine main repo, fetched from Alpine's CDN rather than the
Ubuntu archive that keeps going dark.
- Dockerfile.seektest now uses maven:3.9-eclipse-temurin-11, which
ships JDK 11 and Maven preinstalled — no apt-get maven step.
This also means the runtime images no longer care about
Acquire::Retries / DEBIAN_FRONTEND / apt-get clean, so those lines
are removed with the apt call they were configuring.
* fix(volume): keep vacuum running past dangling .idx entries
Vacuum compaction aborted entirely on the first .idx entry whose offset
pointed past the end of the .dat file, surfacing as `cannot hydrate
needle from file: EOF` and stalling progress on every other volume.
In both Go and Rust:
- During compaction, skip an unreadable needle and continue. The bytes
it pointed at were already unreachable via reads, so dropping the
index reference makes the post-vacuum volume consistent. Real EIO
still bails out so a disk fault is not silently papered over.
- At volume load, do a single linear scan of the .idx and confirm
every (offset + actual size) fits inside .dat. The pre-existing
integrity check only looked at the last 10 entries, so deeper
corruption (e.g. left over from a crashed batched write) went
undetected and only surfaced later as a vacuum EOF. A failure now
marks the volume read-only at load time so an operator can react.
Refs #8928
* fix(volume): only skip permanent-corruption needle reads during vacuum
Address PR review feedback (gemini-code-assist + coderabbit):
The original patch skipped any non-EIO read failure, which would silently
drop needles on transient errors — Windows hardware bad-sector errors
(ERROR_CRC etc.) never surface as syscall.EIO; tiered-storage network
timeouts and EROFS would also slip through and shrink the volume.
Switch to an explicit whitelist of permanent-corruption shapes:
- Add needle.ErrorCorrupted sentinel and wrap CRC and "index out of
range" errors with %w so callers can match via errors.Is.
- copyDataBasedOnIndexFile now skips only when the read failure is
io.EOF, io.ErrUnexpectedEOF, ErrorSizeMismatch, ErrorSizeInvalid,
or ErrorCorrupted. Anything else (real disk faults, environmental
errors, Windows hardware codes) aborts the compaction so an
operator notices.
- Mirror the same whitelist in the Rust volume server, matching on
io::ErrorKind::UnexpectedEof and the NeedleError corruption variants
(SizeMismatch, CrcMismatch, IndexOutOfRange, TailTooShort).
Also add `defer v.Close()` in TestVerifyIndexFitsInDat so Windows
t.TempDir() cleanup can release the .dat/.idx handles.
Refs #8928
* fix(volume): wrap entry-not-found size-mismatch with ErrorSizeMismatch
Address PR review: the fallback branch in ReadBytes returned an
unwrapped fmt.Errorf, so isSkippableNeedleReadError (and any caller
using errors.Is(..., ErrorSizeMismatch)) could not match it. Wrap
with %w so the whitelist applies, while leaving the existing direct
sentinel return for the OffsetSize==4 / offset<MaxPossibleVolumeSize
retry path unchanged so ReadData's `err == ErrorSizeMismatch` retry
still triggers.
Refs #8928
* fix(volume): integrate dangling-idx check into existing index load walk
Address PR review (gemini-code-assist, medium): the structural .idx
check used to do a second linear scan of the index file at every volume
load, doubling the disk-I/O cost on servers managing many volumes.
Track the largest (offset + actual size) seen during the existing
needle-map load walks (`LoadCompactNeedleMap`, `NewLevelDbNeedleMap`,
`NewSortedFileNeedleMap`'s `newNeedleMapMetricFromIndexFile`,
`DoOffsetLoading`) on a new `MaximumNeedleEnd` field on `mapMetric`,
exposed as `MaxNeedleEnd()` on the NeedleMapper interface.
`volume.load()` then compares `nm.MaxNeedleEnd()` to the .dat size
after the load is complete — pure numeric comparison, no extra I/O.
The standalone `verifyIndexFitsInDat` helper and its caller in
`CheckVolumeDataIntegrity` are removed; the test that used to drive
the helper directly now exercises the new path via
`LoadCompactNeedleMap`.
Mirror the same change in the Rust volume server: track
`max_needle_end` on `NeedleMapMetric`, expose via `max_needle_end()`
on `CompactNeedleMap`, `RedbNeedleMap`, and the `NeedleMap` enum.
The Rust load walk already happens in `load_from_idx` for both map
kinds, so the structural check becomes free.
Refs #8928
Multiple master pushes within the same minute produced identical BUILD_TIME
values, causing concurrent workflow runs to race on identically-named release
assets. Upload retries hit 422 already_exists and failed the build. Adding a
concurrency group with cancel-in-progress ensures only the latest dev build
runs at a time, which is fine since only the latest dev artifacts matter.
* fix(mini): shut down admin/s3/webdav/filer before volume/master on Ctrl+C
Interrupts fired grace hooks in registration order, so master (started
first) shut down before its clients, producing heartbeat-canceled errors
and masterClient reconnection noise during weed mini shutdown. Admin/s3/
webdav had no interrupt hooks at all and were killed at os.Exit.
- grace: execute interrupt hooks in LIFO (defer-style) order so later-
started services tear down first.
- filer: consolidate the three separate interrupt hooks (gRPC / HTTP /
DB) into one that runs in order, so filer shutdown stays correct
independent of FIFO/LIFO semantics.
- mini: add MiniClientsShutdownCtx (separate from test-facing
MiniClusterCtx) plus an OnMiniClientsShutdown helper. Admin, S3,
WebDAV and the maintenance worker observe it; runMini registers a
cancel hook after startup so under LIFO it fires first and waits up to
10s on a WaitGroup for those services to drain before filer, volume,
and master shut down.
Resulting order on Ctrl+C: admin/s3/webdav/worker -> filer (gRPC -> HTTP
-> DB) -> volume -> master.
* refactor(mini): group mini-client shutdown into one state struct
The first pass spread the shutdown plumbing across three globals
(MiniClientsShutdownCtx, miniClientsWg, cancelMiniClients) and two
ctx-derivation sites (OnMiniClientsShutdown and startMiniAdminWithWorker).
Group into a private miniClientsState (ctx/cancel/wg) rebuilt per runMini
invocation, and chain its ctx from MiniClusterCtx so clients only observe
one signal. Tests that cancel MiniClusterCtx still trigger client
shutdown via parent-child propagation.
- resetMiniClients() installs fresh state at the top of runMini, so
in-process test reruns don't inherit stale ctx/wg.
- onMiniClientsShutdown(fn) replaces the exported OnMiniClientsShutdown
and only observes one ctx.
- trackMiniClient() replaces the manual wg.Add/Done dance for the admin
goroutine.
- miniClientsCtx() gives the admin startup a ctx without re-deriving.
- triggerMiniClientsShutdown(timeout) is the interrupt hook body.
No behaviour change; existing tests pass.
* refactor: generalize shutdown ctx as an option, not a mini-specific helper
Several service files (s3, webdav, filer, master, volume) observed the
mini-specific MiniClusterCtx or called onMiniClientsShutdown directly.
That leaked mini orchestration into code that also runs under weed s3,
weed webdav, weed filer, weed master, and weed volume standalone.
Replace with a generic `shutdownCtx context.Context` field on each
service's Options struct. When non-nil, the server watches it and shuts
down gracefully; when nil (standalone), the shutdown path is a no-op.
Mini wires the contexts up from a single place (runMini):
- miniMasterOptions/miniOptions.v/miniFilerOptions.shutdownCtx =
MiniClusterCtx (drives test-triggered teardown)
- miniS3Options/miniWebDavOptions.shutdownCtx = miniClientsCtx() (drives
Ctrl+C teardown before filer/volume/master)
All knowledge of MiniClusterCtx now lives in mini.go.
* fix(mini): stop worker before clients ctx so admin shutdown isn't blocked
Symptom on Ctrl+C of a clean weed mini: mini's Shutting down admin/s3/
webdav hook sat for 10s then logged "timed out". Admin had started its
shutdown but was blocked inside StopWorkerGrpcServer's GracefulStop,
waiting for the still-connected worker stream. That in turn left filer
clients connected and cascaded into filer's own 10s gRPC graceful-stop
timeout.
Two causes, both fixed:
1. worker.Stop() deadlocked on clean shutdown. It sent ActionStop (which
makes managerLoop `break out` and exit), then called getTaskLoad()
which sends to the same unbuffered cmd channel — no receiver, hangs
forever. Reorder Stop() to snapshot the admin client and drain tasks
BEFORE sending ActionStop, and call Disconnect() via the local
snapshot afterwards.
2. Worker's taskRequestLoop raced with Disconnect(): RequestTask reads
from c.incoming, which Disconnect closes, yielding a nil response and
a panic on response.Message. Handle the closed channel explicitly.
3. Mini now has a preCancel phase (beforeMiniClientsShutdown) that runs
synchronously BEFORE the clients ctx is cancelled. Register worker
shutdown there so admin's worker-gRPC GracefulStop finds the worker
already disconnected and returns immediately, instead of waiting on
a stream that is about to close anyway.
Observed shutdown of a clean mini: admin/s3/webdav down in <10ms; full
process exit in ~11s (the remaining 10s is a pre-existing filer gRPC
graceful-stop timeout, not cascaded from the clients tier).
* feat(mini): cap filer gRPC graceful stop at 1s under weed mini
Full weed mini shutdown was ~11s on a clean exit, dominated by the
filer's default 10s gRPC GracefulStop timeout while background
SubscribeLocalMetadata streams drained.
Expose the timeout as a FilerOptions.gracefulStopTimeout field (default
10s for standalone weed filer) and set it to 1s in mini. Clean weed mini
shutdown now takes ~2s.
* fix(iceberg): route catalog clients to the right bucket and vend S3 endpoint
DuckDB ATTACH 's3://<bucket>/' AS cat (TYPE 'ICEBERG', ...) was failing
with "schema does not exist" because GET /v1/config ignored the warehouse
query parameter and returned no overrides.prefix, so subsequent requests
fell through to the hard-coded "warehouse" default bucket instead of the
one the client attached. LoadTable also returned an empty config, forcing
clients to discover the S3 endpoint out-of-band and producing 403s on
direct iceberg_scan calls.
- handleConfig now echoes overrides.prefix = bucket and defaults.warehouse
when ?warehouse=s3://<bucket>/ is supplied.
- getBucketFromPrefix honors a warehouse query parameter as a fallback for
clients that skip the /v1/config handshake.
- LoadTable responses advertise s3.endpoint and s3.path-style-access so
clients can reach data files without separate configuration.
Refs #9103
* address review feedback on iceberg S3 endpoint vending
- deriveS3AdvertisedEndpoint is now a method on S3Options; honors
externalUrl / S3_EXTERNAL_URL, switches to https when -s3.key.file is
set, uses the https port when configured, and brackets IPv6 literals
via util.JoinHostPort.
- handleCreateTable returns s.buildFileIOConfig() in both its staged and
final LoadTableResult branches so create and load flows see the same
FileIO hints.
- Add unit test coverage for the endpoint derivation scenarios.
* address CI and review feedback for #9109
- DuckDB integration test now runs under its own newOAuthTestEnv (with a
valid IAM config) so the OAuth2 client_credentials flow DuckDB requires
actually works; the shared env has no registered credentials, which was
the cause of the CI failure. Helper createTableWithToken was added to
create tables via Bearer auth.
- Tighten TestIssue9103_LoadTableDoesNotVendS3FileIOCredentials to also
assert s3.path-style-access = "true", so a partial regression where the
endpoint is vended but path-style is dropped still fails.
- deriveS3AdvertisedEndpoint now logs a startup hint when it infers the
host from os.Hostname because the bind IP is a wildcard, pointing
operators at -s3.externalUrl / S3_EXTERNAL_URL for reverse-proxy
deployments where the inferred name is not externally reachable.
- handleConfig has a comment explaining that any sub-path in the
warehouse URL is dropped because catalog routing is bucket-scoped.
* fix(iceberg): make advertised S3 endpoint strictly opt-in; add region
The wildcard-bind fallback to os.Hostname() in deriveS3AdvertisedEndpoint
was hijacking correctly-configured clients: on the CI runner it produced
http://runnervmrc6n4:<port>, which Spark (running in Docker) could not
resolve, so Spark iceberg tests began failing after the endpoint started
being vended in LoadTable responses.
Change the rule so advertising is opt-in and never guesses a host that
might not be routable:
- -s3.externalUrl / S3_EXTERNAL_URL wins (covers reverse-proxy).
- Otherwise, only an explicit, non-wildcard -s3.bindIp is used.
- Wildcard / empty bind returns "" so no FileIO endpoint is vended and
existing clients keep using their own configuration.
buildFileIOConfig additionally vends s3.region (defaulting to the same
value baked into table bucket ARNs) whenever it vends an endpoint, so
DuckDB's attach does not fail with "No region was provided via the
vended credentials" when the operator has opted in.
The DuckDB issue-9103 integration test runs under an env with a
wildcard bind, so it explicitly sets AWS_REGION in the docker run to
pick up the same default. The HTTP-level LoadTable-vending test was
dropped because its expectation is now conditional and already covered
by unit tests in iceberg_issue_9103_test.go.
* fix(mount): remove fid pool to stop master over-allocating volumes
The writeback-cache fid pool pre-allocated file IDs with
ExpectedDataSize = ChunkSizeLimit (typically 8+ MB). The master's
PickForWrite charges count * expectedDataSize against the volume's
effectiveSize, so a full pool refill could charge hundreds of MB
against a single volume before any bytes were actually written.
That tripped RecordAssign's hard-limit path and eagerly removed
volumes from writable, causing the master to grow new volumes
even when the real data being written was tiny.
Drop the pool entirely. Every chunk upload goes through
UploadWithRetry -> AssignVolume with no ExpectedDataSize hint,
letting the master fall back to the 1 MB default estimate. The
mount->filer grpc connection is already cached in pb.WithGrpcClient
(non-streaming mode), so per-chunk AssignVolume is a unary RPC
over an existing HTTP/2 stream, not a full dial. Path-based
filer.conf storage rules now apply to mount chunk assigns again,
which the pool had to skip.
Also remove the now-unused operation.UploadWithAssignFunc and its
AssignFunc type.
* fix(upload): populate ExpectedDataSize from actual chunk bytes
UploadWithRetry already buffers the full chunk into `data` before
calling AssignVolume, so the real size is known. Previously the
assign request went out with ExpectedDataSize=0, making the master
fall back to the 1 MB DefaultNeedleSizeEstimate per fid — same
over-reservation symptom the pool had, just smaller per call.
Stamp ExpectedDataSize = len(data) before the assign RPC when the
caller hasn't already set it. This covers mount chunk uploads,
filer_copy, filersink, mq/logstore, broker_write, gateway_upload,
and nfs — all the UploadWithRetry paths.
* fix(assign): pass real ExpectedDataSize at every assign call site
After removing the mount fid pool, per-chunk AssignVolume calls went
out with ExpectedDataSize=0, making the master fall back to its 1 MB
DefaultNeedleSizeEstimate. That's still an over-estimate for small
writes. Thread the real payload size through every remaining assign
site so RecordAssign charges effectiveSize accurately and stops
prematurely marking volumes full.
- filer: assignNewFileInfo now takes expectedDataSize and stamps it
on both primary and alternate VolumeAssignRequests. Callers pass:
- SSE data-to-chunk: len(data)
- copy manifest save: len(data)
- streamCopyChunk: srcChunk.Size
- TUS sub-chunk: bytes read
- saveAsChunk (autochunk/manifestize): 0 (small, size unknown
until the reader is drained; master uses 1 MB default)
- filer gRPC remote fetch-and-write: ExpectedDataSize = chunkSize
after the adaptive chunkSize is computed.
- ChunkedUploadOption.AssignFunc gains an expectedDataSize parameter;
upload_chunked.go passes the buffered dataSize at the call site.
S3 PUT assignFunc stamps it on the AssignVolumeRequest.
- S3 copy: assignNewVolume / prepareChunkCopy take expectedDataSize;
all seven call sites pass the source chunk's Size.
- operation.SubmitFiles / FilePart.Upload: derive per-fid size from
FileSize (average for batched requests, real per-chunk size for
sequential chunk assigns).
- benchmark: pass fileSize.
- filer append-to-file: pass len(data).
* fix(assign): thread size through SaveDataAsChunkFunctionType
The saveAsChunk path (autochunk, filer_copy, webdav, mount) ran
AssignVolume before the reader was drained, so it had to pass
ExpectedDataSize=0 and fall back to the master's 1 MB default.
Add an expectedDataSize parameter to SaveDataAsChunkFunctionType.
- mergeIntoManifest already has the serialized manifest bytes, so
it passes uint64(len(data)) directly.
- Mount's saveDataAsChunk ignores the parameter because it uses
UploadWithRetry, which already stamps len(data) on the assign
after reading the payload.
- webdav and filer_copy saveDataAsChunk follow the same UploadWithRetry
path and also ignore the hint.
- Filer's saveAsChunk (used for manifestize) plumbs the value to
assignNewFileInfo so manifest-chunk assigns get a real size.
Callers of saveFunc-as-value (weedfs_file_sync, dirty_pages_chunked)
pass the chunk size they're about to upload.
The Link and CopyFileRange FUSE request handlers were calling
fuseServer.InodeNotify (and EntryNotify for copy) synchronously while
the kernel was still waiting for the request's reply on the same
/dev/fuse fd. Notifications share that fd, so the syscall.Write can
block indefinitely when the kernel hasn't drained its queue yet,
hanging the entire mount. A goroutine dump from a stuck mount showed
the Link handler blocked in syscall.Write inside InodeNotify while the
server's read loop kept waiting for new requests.
Drop the synchronous notifies. The local meta cache is still updated
inline, so subsequent filesystem ops see the fresh state; the kernel's
attr/dentry caches re-fetch once their TTL expires.
* fix(master): eagerly remove volume from writable when RecordAssign hits limit
Previously, a volume was only removed from the writable list by the
heartbeat-driven CollectDeadNodeAndFullVolumes pass, which runs every
pulse (5s) after a 5s heartbeat. Under sustained concurrent writes,
fio-style workloads observed in the field grew volumes 8-20x past the
configured 100MB limit (median 530MB, peak 1.98GB) during that
5-15s detection window.
RecordAssign already tracks effective size (reported + pending) on each
/dir/assign. It now also removes the volume from writable the moment
effectiveSize reaches volumeSizeLimit, and mirrors the activeVolumeCount
decrement that Topology.SetVolumeCapacityFull would have done on the
next heartbeat. The heartbeat path remains unchanged and idempotent
(vl.SetVolumeCapacityFull returns false if already removed, so no
double-decrement).
Recovery still works: if a heartbeat later reports size < limit and
the volume is not oversized, EnsureCorrectWritables adds it back.
- weed/topology/volume_layout.go: RecordAssign returns reachedCapacity
bool; adds AdjustActiveVolumeCountForFull helper.
- weed/topology/topology.go: PickForWrite invokes the decrement on
eager full transitions.
- TestPickForWrite: pass a 1024-byte hint instead of 0 so the default
1MB pendingDelta does not immediately bust the test's 32KB limit.
- New TestRecordAssignReachingCapacityRemovesFromWritable covers the
eager removal, active count accounting, and no-double-accounting.
* fix(master): recover eagerly-removed volume once decay clears pending
After RecordAssign eagerly removes a volume from writables because
effectiveSize reached the limit, decay can later bring effectiveSize
back under the limit (e.g., when a burst of assigns didn't all result
in uploads). Without recovery the volume would stay non-writable until
vacuum or a ReadOnly flip.
UpdateVolumeSize now re-adds the volume to writables once all of the
following hold:
* RecordAssign is what removed it (tracked via fullSince timestamp)
* at least capacityRecoveryDelay has elapsed since the removal (30s)
— this prevents bouncing during a steady stream of assigns near
the limit
* effectiveSize has decayed below the crowded threshold (90% of limit)
* reportedSize is under the limit (actual disk is not over)
* standard EnsureCorrectWritables preconditions: enough copies, all
copies writable, not oversized
The caller (SyncDataNodeRegistration) re-increments activeVolumeCount
symmetrically with the decrement done on eager removal.
* review: release VolumeLayout lock before UpAdjustDiskUsageDelta
adjustActiveVolumeCount held vl.accessLock across the tree-climbing
UpAdjustDiskUsageDelta walk. That walk takes per-level DiskUsages
locks and could be re-entered from other call paths that hold a
node-level lock and then acquire vl.accessLock. Copy the node list
under the VolumeLayout lock and release it before the tree walk to
eliminate the lock-ordering hazard.
* ci(pjdfstest): cache docker layers via GHA to avoid apt mirror flakes
Replace the local buildx cache + manual fallback with docker/setup-buildx-action
and docker/build-push-action using type=gha cache. The e2e and pjdfstest Dockerfile
layers now persist across runs in GitHub's own cache backend, so apt-get update
only hits Ubuntu mirrors when the Dockerfiles change. Also add Acquire::Retries
and Timeout so first-run cache-miss builds survive transient mirror sync errors.
* ci(pjdfstest): use local registry to share e2e image across buildx builds
The docker-container buildx driver cannot see images loaded into the host
Docker daemon, so the second build's FROM chrislusf/seaweedfs:e2e failed with
"not found" on registry-1.docker.io.
Run a local registry:2 on the runner, push both images to localhost:5000,
remap the FROM via build-contexts so the Dockerfile stays unchanged, then
tag the pulled images locally for docker compose to consume.
* fix(filer): drop stale master gRPC cache on stream death (#9102)
When the master server restarts behind a stable L4 endpoint (e.g. a
Kubernetes ClusterIP Service), the filer's streaming KeepConnected
channel detects the disconnect and reconnects, but the shared
request-path ClientConn cached in pb.grpcClients can remain in READY
state while actually being dead. New AssignVolume/LookupVolume calls
reuse that cached channel and return `rpc error: code = Canceled desc
= context canceled` for every request, until the filer pod is
restarted.
- Expose pb.InvalidateGrpcConnection(address) to drop a cached
ClientConn when a higher-level signal says it is stale.
- In MasterClient.tryConnectToMaster, invalidate the cached
request-path channel whenever the KeepConnected stream returns, so
unrelated callers dial fresh on their next RPC.
- Extend operation.Assign's retry predicate to cover Canceled and
DeadlineExceeded while the caller context is still live: the first
failure invalidates the stale ClientConn via
shouldInvalidateConnection, and the retry dials a new channel.
* fix(grpc): invalidate cached peer conn on streaming death in other paths
Extends the master-client fix to the other streaming-caller + cached
non-streaming-peer pairs that share the same stale-channel failure mode
when the peer restarts behind a stable L4 endpoint (k8s Service VIP,
external load balancer):
- pb.FollowMetadata (s3, mount, webdav, mq broker, filer remote gateway,
etc. → filer): invalidate the filer's cached ClientConn when the
SubscribeMetadata stream returns an error.
- filer.MetaAggregator.loopSubscribeToOneFiler (filer → peer filer):
invalidate the peer's cached ClientConn after doSubscribeToOneFiler
fails, so the next iteration's readFilerStoreSignature / updateOffset
calls dial fresh.
- mq sub_client.onEachPartition and doKeepConnectedToSubCoordinator
(subscriber → broker): invalidate the broker's cached ClientConn when
the SubscribeMessage / SubscriberToSubCoordinator stream errors.
- mq broker.BrokerConnectToBalancer (broker → broker-balancer):
invalidate the balancer's cached ClientConn after the
PublisherToPubBalancer stream errors.
* address review feedback on InvalidateGrpcConnection
- pb.InvalidateGrpcConnection: drop the cache entry under grpcClientsLock
but call ClientConn.Close() after releasing the lock, so Close's
internal synchronisation/IO doesn't serialise unrelated callers on the
global map lock.
- wdclient.tryConnectToMaster: only invalidate the cached request-path
channel when the streaming call returned an error. On a healthy leader
redirect (gprcErr == nil) the cached channel is still usable and
invalidating it just causes a needless re-dial from concurrent callers.
* refactor(grpc): centralize peer-conn invalidation in streaming path
Previously every streaming caller duplicated the same invalidate-cached-
non-streaming-peer-conn wrapper around their WithGrpcClient(true, ...)
call. Move that logic into WithGrpcClient itself: when the streaming
fn returns an error, invalidate any cached ClientConn for the same
address. This removes six near-identical call-site wrappers and gives
every current and future streaming caller the fix by default.
Also aligns the non-streaming branch with the new Invalidate helper's
lock discipline: delete the cache entry under grpcClientsLock, then
Close the ClientConn after releasing the lock.
* feat(mount): set FOPEN_KEEP_CACHE when file mtime is unchanged
On re-open of an unmodified file, signal the kernel to preserve its
existing page cache. This eliminates redundant volume server reads for
workloads that repeatedly open-read-close the same files (build systems,
config readers, etc.).
* fix(mount): use guarded type assertion for openMtimeCache load
Use the two-value form of type assertion when loading from sync.Map
to prevent potential panics if a non-int64 value is ever stored.
* fix(mount): skip redundant mtime store and invalidate on truncation
- Avoid redundant sync.Map Store when cached mtime already matches
the current mtime, reducing contention on the hot open path.
- Invalidate openMtimeCache in SetAttr when file size changes
(truncation), preventing stale kernel page cache after ftruncate.
* fix(mount): use nanosecond mtime precision and bounded cache for FOPEN_KEEP_CACHE
- Compare both Mtime (seconds) and MtimeNs (nanoseconds) to detect
sub-second modifications common in automated workloads.
- Replace unbounded sync.Map with a bounded map + mutex (8192 entries,
random eviction when full), following the existing atimeMap pattern.
- Extract applyKeepCacheFlag and invalidateOpenMtimeCache methods for
clarity and testability.
- Add tests for nanosecond precision and cache eviction.
* fix(mount): invalidate mtime cache in truncateEntry for O_TRUNC consistency
Add invalidateOpenMtimeCache call to truncateEntry so the Create path
with O_TRUNC follows the same explicit invalidation pattern as SetAttr
and Write.
The consumer group resumption flow in TestOffsetManagement occasionally fails
with `read tcp ... i/o timeout` on the second consumer's first FetchMessage.
Re-joining an existing group races with the previous member's LeaveGroup and
session cleanup; the new reader can observe transient coordinator state that
the kafka-go client surfaces as a connection read timeout.
Wrap ConsumeWithGroup in a 3-attempt retry that rebuilds the reader only when
no message was received yet, so a transient join failure doesn't fail the
whole test. Partial progress (at least one message consumed) still returns
immediately, preserving the original error semantics for post-join failures.
* perf(mount): add graduated write backpressure before buffer cap
Introduce soft (80%) and hard (95%) throttling thresholds in
WriteBufferAccountant. When write buffer usage approaches the cap,
Reserve() inserts brief sleeps to slow writers gradually rather than
blocking them completely at the cap. This smooths out write latency
under sustained load.
* fix(mount): address review feedback for graduated backpressure
- Use projected usage (used + n) for threshold checks so a single
reservation crossing a threshold is throttled immediately.
- Widen no-throttle test timing tolerance to softThrottleDelay to
avoid CI flakes from scheduling jitter.
- Replace fragile timing upper-bound in recovery test with
counter-based invariant (hardThrottleCount stays at 1).
* docs(mount): clarify single-shot throttle design in WriteBufferAccountant
Add a comment explaining why graduated throttling runs once per Reserve
call rather than inside the blocking loop: once at the cap, the evictor
+ cond.Wait mechanism frees actual capacity, which time-based sleeps
cannot do.
* feat(mount): add singleflight deduplication for concurrent chunk reads
When multiple FUSE readers request the same uncached chunk concurrently,
only one network fetch is performed. Other readers wait and share the
downloaded data, reducing redundant volume server traffic under parallel
read workloads.
* fix(util): make singleflight panic-safe with defer cleanup
If the provided function panics, the WaitGroup and map entry are now
cleaned up via defer, preventing other waiters from hanging forever.
* fix(filer): remove singleflight from reader_cache to fix buffer ownership
The singleflight wrapper around chunk fetches returned the same []byte
buffer to concurrent callers. Since each SingleChunkCacher owns and
frees its data buffer in destroy(), sharing the same slice would cause
a use-after-free or double-free with the mem allocator.
The downloaders map already deduplicates in-flight downloads for the
same fileId, so the singleflight was redundant at this layer. The
SingleFlightGroup utility is retained for use elsewhere.
* perf(cache): drop OS page cache after disk cache reads
After reading from the on-disk chunk cache, advise the kernel via
FADV_DONTNEED to release the corresponding page cache pages. This
prevents double-caching the same data in both user-space and kernel
page caches, freeing RAM for other uses on systems with large disk
caches.
* fix(cache): guard dropReadCache against zero length and invalid fd
A zero-length fadvise is interpreted as "to end of file" on Linux,
which would inadvertently drop the page cache for the entire remainder
of the cache volume. Also check fd >= 0 to avoid unnecessary syscalls
when the backend file is closed.
* perf(cache): only apply FADV_DONTNEED for reads >= 1 MiB
For small needle reads the syscall overhead outweighs the memory
savings, and the kernel page cache is more beneficial for warm data.
Restrict fadvise to reads of at least 1 MiB where the freed page
cache is meaningful.
* fix(mount): count manifest sizes in merge condition to prevent manifest accumulation
shouldMergeChunks only counted non-manifest chunk sizes toward the bloat
threshold, so overlapping manifests accumulated undetected across flush
cycles.
During sustained random writes (e.g. fio), each metadata flush compacts
non-manifest chunks and may group them into a new manifest via
MaybeManifestize, while carrying forward all existing manifests. Since
the merge condition only checked non-manifest totals against 2x file
size, the growing pile of redundant manifests never triggered a merge.
In a real workload: 4 GB file, 25 manifests each covering ~4 GB
(107 GB manifest data on volume servers), but shouldMergeChunks saw only
4.2 GB of non-manifest chunks vs the 8.6 GB threshold -- no merge.
Fix: include manifest coverage sizes in totalChunkSize. This correctly
detects the 111 GB total vs 8.6 GB threshold and triggers the merge,
which re-reads the file as clean chunks and lets the filer's MinusChunks
(which resolves manifests) garbage-collect all redundant sub-chunks.
* test(mount,filer): add manifest chunk correctness tests
Filer-level tests (filechunk_manifest_test.go):
- TestManifestRoundTripPreservesChunks: create -> serialize -> deserialize
preserves fileId, offset, size, and timestamp for all sub-chunks
- TestCompactResolvedOverlappingManifests: two manifests covering the same
range, older sub-chunks correctly identified as garbage after compaction
- TestDoMinusChunksWithResolvedManifests: old manifest sub-chunks detected
as garbage when compared against new clean chunks (mirrors filer cleanup)
- TestManifestizeSmallBatchWithRemainder: batch=3 with 7 chunks produces
2 manifests + 1 remainder, all data recoverable after resolve
- TestCompactMultipleOverlappingManifestGenerations: 5 generations of
full-file manifests, compaction keeps only the newest generation
- TestManifestBloatDetection: with N overlapping manifests, merge triggers
at N >= 2 (stored > 2x file size)
Mount-level test (weedfs_file_sync_test.go):
- TestFlushCycleManifestAccumulation: simulates the flushMetadataToFiler
pipeline over multiple cycles with a small manifest batch (5 chunks).
Verifies merge triggers at cycle 2 when accumulated manifests exceed
the 2x threshold. Uses SeparateManifestChunks + CompactFileChunks +
shouldMergeChunks to mirror the real code path.
* fix(mount): reduce chunk fragmentation from random writes
Random writes via FUSE mount create many small partially-overlapping
chunks that bloat storage and slow reads. Two complementary fixes:
1. Fill gaps at seal time: when a partial writable chunk is sealed for
upload, read existing file data into the unwritten regions so
SaveContent uploads one complete chunk instead of many fragments.
No overhead for sequential writes (IsComplete short-circuits).
2. Merge on fsync: after CompactFileChunks, if total non-manifest
chunk data exceeds 2x the logical file size, re-read and re-upload
the entire file as properly-sized chunks. The filer's cleanupChunks
garbage-collects all superseded chunks.
* revert FillGaps: reading from volume servers during write path is too expensive
* test(mount): add tests for chunk merge condition
Extract shouldMergeChunks as a testable function and add tests covering:
- empty file, single chunk, non-overlapping chunks (no merge)
- exactly 2x boundary (no merge) vs just over 2x (merge)
- manifest chunks extend file size but don't count toward stored total
- input slices are not mutated (safe append)
- CompactFileChunks + condition: fully superseded, staggered overlaps,
75% overlap pattern, many 4K writes at 1K step
- randomized bloat detection (50 iterations)
- visible content preserved after compaction (100 iterations)
* fix(mount): cap merge buffer at file size for small files
Avoids allocating a full ChunkSizeLimit buffer when the file being
merged is smaller. Addresses PR review feedback.
* fix(plugin): remove Min Volume Age field from vacuum plugin worker config
The min_volume_age_seconds setting is not needed as a user-configurable
field in the plugin worker form. The detection logic continues to use
the hardcoded default from NewDefaultConfig().
* fix(plugin): disable volume age filtering in plugin worker detection
Set MinVolumeAgeSeconds to 0 in deriveVacuumConfig so the plugin worker
does not filter out volumes by age during detection.
* fix(plugin): remove all volume age filtering from plugin worker detection
Remove age-related fields from detection trace messages, activity
reports, and per-volume diagnostics. The plugin worker now only
filters by garbage threshold.
* feat(mount): proactive flush of idle writable chunks
Add a background goroutine that periodically scans writable chunks
across all open file handles and seals those that are idle and
unlikely to receive further writes, submitting them for async upload.
A writable chunk is proactively flushed when it has been idle for
500ms AND meets one of: nearly full (>=90%), behind the sequential
write frontier by 2+ chunks, or stale for 5+ seconds. Flushing only
happens when the upload pipeline has spare capacity (< half of
concurrent writer slots in use).
This prevents partial chunks from accumulating until fsync/close,
which is particularly beneficial for bursty or small-file workloads
where chunks may never reach IsComplete().
Also fixes a latent bug in ActivityScore where MarkRead/MarkWrite
used value receivers, silently discarding all mutations.
* refactor(mount): reuse WriterPattern instead of duplicating sequential detection
Remove the isSequential atomic from UploadPipeline. The proactive
flusher now reads IsSequentialMode() from the existing WriterPattern
on PageWriter and passes it as a parameter to ProactiveFlush. This
avoids duplicating the sequential/random detection that WriterPattern
already maintains.
* fix(mount): address PR review feedback
- Make ActivityScore thread-safe using atomics (CAS loop for score
updates, atomic load/swap for timestamp). Previously MarkRead was
called under RLock while MarkWrite held a write lock, creating a
data race on the shared fields.
- Fix ProactiveFlush half-capacity guard: use multiplication
(uploaderCount*2 >= max) instead of floor division (max/2) which
misbehaves for odd or small concurrentWriterMax values.
* fix(mount): review fixes for proactive flush
- Fix TOCTOU race in lastWriteChunkIndex update: use CAS loop so
concurrent writers cannot regress the frontier.
- Remove unused UploaderCount() getter.
- Reuse the caller-provided tsNs instead of calling time.Now() again
in WriteDataAt for lastWriteTsNs, eliminating a redundant syscall
per write.
* fix(admin): list all masters and dedupe EC file counts in dashboard
Dashboard -> Master Nodes only ever showed the currently connected master
because getMasterNodesStatus hard-coded a single entry. Replace it with a
RaftListClusterServers call that returns every master in the raft group and
tags the real leader, falling back to the current master only if the raft
call fails.
Buckets -> Object Store Buckets could render 0 objects for a bucket backed
by an EC volume. Every shard holder reports the same whole-volume
file_count (read from the replicated .ecx), so the first-seen value wins;
if that first node had not yet finished loading .ecx it reported 0 and
pinned the aggregate at 0. Take the max across reporting nodes instead.
The dashboard header total_files also dropped after volumes were converted
to erasure coding because getTopologyViaGRPC never folded EC file_count
into topology.TotalFiles. Aggregate it with the same max/sum dedupe.
* fix(admin): address PR review comments
- bound RaftListClusterServers with a 3s timeout so the dashboard endpoint
cannot hang on a stalled master
- pre-validate raft addresses with net.SplitHostPort before calling
pb.GrpcAddressToServerAddress, which otherwise glog.Fatalf's on a
malformed entry and would crash the admin process
- when raft is unreachable, mark the fallback master as not-leader rather
than claiming leadership the code cannot verify
- warn when summed EC delete_count exceeds file_count while folding into
topology.TotalFiles, matching collectCollectionStats
* fix(admin): distinguish empty raft response from RPC failure
When RaftListClusterServers returns successfully with no servers, raft is
not initialized (standalone/non-raft cluster), so the single fallback
master is the leader. Only treat the fallback as a non-leader when the
RPC actually failed.
* fix(admin): remove misleading Objects column from S3 buckets page
The bucket "Objects" column displayed needle counts from volume
collection stats, not actual S3 object counts. This is confusing
because a single S3 object can span multiple needles (multipart
uploads, versions) and the count is inaccurate for EC volumes.
Remove the ObjectCount field from S3Bucket, the Objects table column,
the sort-by-objects handler, the detail-view row, and both CSV export
references.
* fix(admin): correct cell indexes in fallback bucket CSV export
After the Objects column was removed, the fallback CSV exporter in
admin.js still used stale cell indexes: cells[1] mapped to Owner
(not Created), cells[2] to Created (not Size), cells[3] to Logical
Size (not Quota). Align all indexes with the current table column
order and include Owner, Logical Size, and Physical Size.
* fix(master): do not re-enter warmup when a fresh cluster grows its first volume
Follow-up investigation on #8777. After fixing the writable-chunk cap
deadlock, a second issue surfaced on the same fio reproducer: the
weed mount would log thousands of
upload data X: filerGrpcAddress assign volume: assign volume failure
count:1 path:"/test2/X": assign volume: rpc error: code = Canceled
desc = grpc: the client connection is closing
cascading from the filer's handler. The mount's own cached gRPC
connection to the filer was never invalidated — the "client connection
is closing" text is the FILER's cached connection to the MASTER going
away, and the message is forwarded verbatim in the filer's
AssignVolumeResponse.Error.
Root cause: Topology.IsWarmingUp checks the *live* GetMaxVolumeId. On
a fresh cluster the master is initialized with SetLastLeaderChangeTime
(master_server.go:239 "Seed the warmup timestamp so IsWarmingUp() is
active even if the leader change event hasn't fired yet"), but the
MaxVolumeId==0 guard is supposed to short-circuit IsWarmingUp for
bootstraps so there is no wait. That guard breaks the moment the
first volume is grown inside the warmup window: MaxVolumeId flips
from 0 to 1, the lastLeaderChangeTime is still within 3*pulse (15 s
default), and IsWarmingUp retroactively returns true for the next
several seconds. Every AssignVolume in that window returns
codes.Unavailable, which trips the filer's shouldInvalidateConnection
guard and tears down its cached master connection, which in turn
surfaces as "client connection is closing" to every concurrent
in-flight call from the mount's file-close flush storm. fio reports
EIO on close and the user sees a thousand scary error lines in the
mount log for an otherwise correct run.
Fix: snapshot `hadVolumesAtLeaderChange` inside SetLastLeaderChangeTime
and read that snapshot in IsWarmingUp instead of the live MaxVolumeId.
A fresh cluster snapshots "no volumes at leader change" → IsWarmingUp
stays false through the entire warmup window regardless of how fast
the first grow lands. A real leader transition on a populated cluster
still snapshots "has volumes" → IsWarmingUp behaves exactly as before
until the 3*pulse window closes. The lock ordering in
SetLastLeaderChangeTime reads MaxVolumeId before taking the
lastLeaderChangeTimeLock so the two calls cannot interleave weirdly;
IsWarmingUp reads both fields under a single RLock acquisition.
Verified with the same containerized reproducer used for the deadlock
fix: 4 jobs × 250 nrfiles × 40 MiB × 4k randwrite direct.
- baseline (master): fio rc=1 (EIO on close), 2809 mount error
lines matching "filerGrpcAddress assign volume ... client
connection is closing", 788 filer lines matching "warming up",
331 "Removing cached gRPC connection to ...19333 due to error:
master is warming up".
- patched: fio rc=0 in 1.9 s at 79.2 MiB/s, 0 mount errors,
0 filer "warming up" lines, 0 cached-master-conn invalidations.
go test ./weed/topology/... passes.
* fix(topology): make NodeImpl.maxVolumeId atomic
CodeRabbit flagged on #9092 that my new IsWarmingUp snapshot
(hadVolumesAtLeaderChange) reads through GetMaxVolumeId(), which until
now returned an unprotected int field on NodeImpl. UpAdjustMaxVolumeId
is called from the volume server heartbeat path in parallel with
GetMaxVolumeId reads on the assign path, and neither side had any
synchronization — a long-standing data race the race detector would
flag if the warmup test suite stressed both sides.
Switch maxVolumeId to atomic.Uint32 (needle.VolumeId is uint32) and
implement UpAdjustMaxVolumeId as a CAS loop so the check-then-set
stays linearizable: two heartbeats racing to promote the field will
land the higher value deterministically and propagate to the parent
exactly once. GetMaxVolumeId is a single atomic load. Callers of both
helpers are unchanged; the struct field comment documents why the
atomic is necessary.
go test -race ./weed/topology/... passes.
* refactor(topology): use WarmupDuration helper in IsWarmingUp
Gemini review on #9092 flagged that IsWarmingUp re-derives the
warmup duration from pulse and WarmupPulseMultiplier instead of
using the existing WarmupDuration() helper, which RemainingWarmupDuration
already uses. Fold the duration calculation through the helper and
short-circuit on lastChange.IsZero() before the time.Since call.
No behavior change.
* fix(mount): evict writable chunks when writeBufferSizeMB cap is reached
Reported on #8777: fio 4k randwrite with --nrfiles=1000 on a weed mount
hangs indefinitely after ~1 second of activity, volume server QPS
drops to zero, and the test never makes progress beyond a few hundred
writes. A scaled-down local repro (4 jobs x 250 nrfiles x 40 MiB with
-chunkSizeLimitMB=32 -writeBufferSizeMB=10240) reproduced the hang
exactly: fio stuck in fio_state=SETTING_UP, mount idle, all writer
goroutines parked in page_writer.(*WriteBufferAccountant).Reserve
with n=0x2000000 (32 MiB).
Root cause: UploadPipeline.SaveDataAt reserves a full chunkSize slot
against the global WriteBufferAccountant the first time it allocates
a writable chunk for a given file. The reservation is released by
SealedChunk.FreeReference only after the chunk's async upload
completes, and chunks only move from writable to sealed when they
fill or when the file is flushed/closed. For 4k random writes with
small per-file data and iodepth holding files open, writable chunks
never fill and never close, so their slots are pinned forever. Once
openFiles * chunkSize exceeds writeBufferSizeMB, every new writer
blocks in Reserve's cond.Wait with no possible waker, a hard deadlock.
The user-visible symptoms (volume QPS=0, waited 30min, still running)
are exactly that state. In the reported config, 8 jobs * 1000 nrfiles
* 32 MiB = 256 GB of reservations against a 10 GB cap.
Fix: add a SetEvictor hook on WriteBufferAccountant. When Reserve
would otherwise block on the cond, it single-flights an evictor
callback that walks wfs.fhMap, picks the first open file handle with
a writable chunk, and force-seals its fullest writable chunk via a
new UploadPipeline.EvictOneWritableChunk. Force-sealing submits the
chunk for async upload on the existing uploader pool; the upload's
SealedChunk.FreeReference releases the global slot, broadcasts on
the accountant cond, and unblocks the waiter. The fullest-first
heuristic matches the existing over-limit path in SaveDataAt and
keeps us from thrashing on repeatedly re-sealing half-empty chunks.
The original #9066 motivation (cap global write-pipeline growth so
swap can't be filled while uploads stall) is preserved: Reserve
still blocks when the cap is actually exhausted by in-flight
uploads, and a failed evictor (no writable chunks to seal) falls
through to cond.Wait exactly as before.
Verified end-to-end in a privileged debian container running a
full master + volume + filer + weed mount stack. Against the
reporter's config shrunk to the same memory footprint (4 jobs x
250 nrfiles x 40 MiB, -chunkSizeLimitMB=32 -writeBufferSizeMB=10240):
- baseline binary (master): fio stalls after ~320 writes, killed
by timeout at 400s with io=1280 KiB and bw=2383 B/s.
- patched binary: fio completes in 1.9 s at 82.8 MiB/s, all
40,000 writes issued, 156 MiB written, rc=0.
page_writer unit tests still pass.
Touches: WriteBufferAccountant (SetEvictor + single-flight in
Reserve), UploadPipeline.EvictOneWritableChunk, DirtyPages
interface + the two existing implementers (ChunkedDirtyPages,
PageWriter), and a new weedfs_write_buffer_evict.go holding the
WFS.evictOneWritableChunk callback that walks fhMap.
* fix(mount): re-check write-budget cap after evict and make eviction panic-safe
Addresses two review concerns on #9091:
1. CodeRabbit flagged that the original Reserve loop only considered the
break condition when the evictor reported success:
if evicted {
if a.used+n <= a.cap || a.used == 0 {
break
}
}
a.cond.Wait()
A concurrent Release firing during the evictor's execution window (we
drop a.mu around the callback so uploader goroutines can drain) would
land its Broadcast on an empty waiter set and then be missed, because
we unconditionally call cond.Wait() on the same iteration. In practice
an in-flight async upload eventually fires another broadcast so this
would delay rather than permanently hang — but the logic is wrong
either way and should re-check the cap after every evictor round.
2. Gemini flagged that setting `a.evicting = true` and dropping `a.mu`
without a deferred unwind leaves the accountant in a broken state if
the evictor panics: the flag stays set forever and no Reserve caller
can ever run another eviction, even though the mutex gets unlocked by
the normal stack unwind.
Pull the evictor call into a tiny helper runEvictorLocked whose defer
re-acquires a.mu and clears a.evicting regardless of panic status, then
broadcasts so other blocked Reservers re-evaluate. Reserve checks the
cap condition immediately after the helper returns; only falls through
to cond.Wait() when the budget is still exhausted. No behavior change on
the happy path; the fix is in the error and race corners.
* doc(mount): clarify eviction-scan cost on the blocked-Reserve path
Gemini review on #9091 flagged that the fhMap scan inside
evictOneWritableChunk is O(open files). Document why the cost is
bounded to the Reserve-blocked path and paid at most once per
chunkSize drained, rather than on the write hot path, and record the
reason we do not preempt with an LRU.
* fix(balance): guard against destination overshoot and oscillation
Plugin-worker volume_balance detection re-selects maxServer/minServer
each iteration based on utilization ratio. With heterogeneous
MaxVolumeCount values, a single greedy move can flip which server is
most-utilized, causing A->B, B->A oscillation within one detection
cycle and pushing destinations past the cluster ideal.
Mirror the shell balancer's per-move guard
(weed/shell/command_volume_balance.go:440): before scheduling a move,
verify that the destination's post-move utilization would not strictly
exceed the source's post-move utilization. If it would, no single move
can improve balance, so stop.
Add regression tests that cover:
- TestDetection_HeterogeneousMax_NoOvershootNoOscillation: 2 servers
with different caps just above threshold; detection must not
oscillate or make the imbalance worse.
- TestDetection_RespectsClusterIdealUtilization: 3-server heterogeneous
layout; destinations must not overshoot cluster ideal.
* fix(balance): use effective capacity when resolving destination disk
resolveBalanceDestination read VolumeCount directly from the topology
snapshot, which is not updated when AddPendingTask registers a move
within the current detection cycle. This meant multiple moves planned
in a single cycle all saw the same static count and could target the
same disk past its effective capacity.
Switch to ActiveTopology.GetNodeDisks + GetEffectiveAvailableCapacity
so that destination planning accounts for all pending and assigned
tasks affecting the disk — consistent with how the detection loop
already tracks effectiveCounts at the server level.
Add a unit test that seeds two pending balance tasks against a
destination disk with 2 free slots and asserts resolveBalanceDestination
rejects a third planned move.
* fix(ec_balance): capacity-weighted guard in Phase 4 global rebalance
detectGlobalImbalance picked min/max nodes by raw shard count and
compared them against a simple (unweighted) rack-wide average. With
heterogeneous MaxVolumeCount across nodes in the same rack, this lets
the greedy algorithm move shards from a large, barely-used node to a
small, nearly-full node just because the small node has fewer shards
in absolute terms — strictly worsening imbalance by utilization and
potentially overfilling the small node.
Snapshot each node's total shard capacity (current shards plus free
slots) at loop start and add a per-move convergence guard: reject any
move where the destination's post-move utilization would strictly
exceed the source's post-move utilization. Mirrors the fix in
weed/worker/tasks/balance/detection.go.
Regression test TestDetectGlobalImbalance_HeterogeneousCapacity covers
a rack with node1 (cap 100, 10 shards → 10% util) and node2 (cap 5,
3 shards → 60% util). Before the fix, Phase 4 moves 2 shards from
node1 to node2, filling node2 to 100% util. After the fix, the guard
blocks both moves.
* fix(ec_balance): utilization-based max/min in Phase 4 rebalance
Phase 4's global rebalancer picked source and destination nodes by raw
shard count, and compared against a simple raw-count average. With
heterogeneous MaxVolumeCount across nodes in a rack, this got the
direction wrong: a large-capacity node holding many shards in absolute
terms but only a small fraction of its capacity would be picked as the
"overloaded" source, while a small-capacity node nearly at its slot
limit (but holding fewer absolute shards) would be picked as the
"underloaded" destination. The previous fix added a strict-improvement
guard that prevented the bad move but left balance untouched — the
rack stayed in an uneven state.
Switch to utilization-based selection and a utilization-based pre-check:
- Pick max/min by (count / capacity), where capacity is the node's
current allowed shards plus remaining free slots (snapshotted once
per rack and held constant for the duration of the loop).
- Replace the raw-count imbalance gate (exceedsImbalanceThreshold) with
a new exceedsUtilImbalanceThreshold helper that compares fractional
fullness. The raw-count gate is still used by Phase 2 and Phase 3,
where the per-rack / per-volume semantics differ.
- Drop the raw-count guards (maxCount <= avgShards || minCount+1 >
avgShards and maxCount-minCount <= 1) now that the per-move
strict-improvement check handles termination correctly for both
homogeneous and heterogeneous capacity.
Also fix a latent bug in the inner shard-selection loop: it was not
updating shardBits between iterations, so every iteration picked the
same lowest-set bit and emitted duplicate move requests for the same
physical shard. Update maxNode and minNode's shardBits immediately
after appending a move, mirroring what applyMovesToTopology does
between phases.
Update TestDetectGlobalImbalance_HeterogeneousCapacity to assert:
- Moves flow from the higher-util node2 to the lower-util node1
(direction check), and
- Each (volumeID, shardID) pair appears at most once in the move list
(duplicate-shard guard).
* fix(ec_balance): keep source freeSlots in sync after planned shard moves
All three phase loops that plan EC shard moves (detectCrossRackImbalance,
detectWithinRackImbalance, detectGlobalImbalance) decrement the
destination node's freeSlots but leave the source node's freeSlots
stale. Over the course of a detection run that processes many volumes
or iterates within a rack, the source's reported freeSlots drifts
below its actual value.
In Phase 4 specifically, the per-move strict-improvement guard prevents
the source from becoming a destination candidate, so the stale value
never affects decisions. In Phases 2 and 3 it can: a node that sheds
shards for one volume's rebalance is eligible as a destination for
another volume in the same run, and the destination selection uses
node.freeSlots <= 0 as a hard skip (findDestNodeInUnderloadedRack /
findLeastLoadedNodeInRack). A tightly-provisioned node could be
skipped as a destination even after it has freed slots.
Increment maxNode.freeSlots / node.freeSlots symmetrically at each
scheduled move so freeSlots remains an accurate running view of
available slot capacity throughout a detection run.
* fix(wdclient,volume): compare master leader with ServerAddress.Equals
Raft leader is advertised as host:httpPort.grpcPort, but clients dial
host:httpPort. Raw string comparison against VolumeLocation.Leader /
HeartbeatResponse.Leader therefore never matches, causing the
masterclient and the volume server heartbeat loop to continuously
"redirect" to the already-connected master, tearing down the stream
and reconnecting.
Use ServerAddress.Equals, which normalizes the grpc-port suffix.
* fix(filer,mq): compare ServerAddress via Equals in two more sites
filer bootstrap skip (MaybeBootstrapFromOnePeer) and the broker's local
partition assignment check both compared a wire-supplied address string
against the local self ServerAddress with raw string equality. Both are
vulnerable to the same plain-vs-host:port.grpcPort mismatch as the
masterclient/volume heartbeat sites: filer would bootstrap from itself,
and the broker would fail to claim a partition it was actually assigned.
Route both through ServerAddress.Equals.
* fix(master,shell): more ServerAddress comparisons via Equals
- raft_server_handlers.go HealthzHandler: s.serverAddr == leader would
skip the child-lock check on the real leader when the two carry
different plain/grpc-suffix forms, returning 200 OK instead of 423.
- master_server.go SetRaftServer leader-change callback: the
Leader() == Name() guard for ensureTopologyId could disagree with
topology.IsLeader() (which already uses Equals), so leader-only
initialization could be skipped after an election.
- command_volume_merge.go isReplicaServer: the -target guard compared
user-supplied host:port against NewServerAddressFromDataNode(...) with
==, letting an existing replica slip through when topology carries
the embedded gRPC port.
All routed through pb.ServerAddress.Equals.
* fix(mq,cluster): more ServerAddress comparisons via Equals
- broker_grpc_lookup.go GetTopicPublishers/GetTopicSubscribers: the
partition ownership check gated listing on raw
LeaderBroker == BrokerAddress().String(), so listings silently omitted
partitions hosted locally when the assignment carried the other
host:port / host:port.grpcPort form.
- lock_client.go: LockHostMovedTo comparison and the seedFiler fallback
guard both used raw string equality against configured filer
addresses (which may be plain host:port while LockHostMovedTo comes
back suffixed), causing spurious host-change churn and blocking the
seed-filer fallback.
* fix(mq): more ServerAddress comparisons via Equals
- pub_balancer/allocate.go EnsureAssignmentsToActiveBrokers: direct
activeBrokers.Get() lookup missed brokers when a persisted assignment
carried a different address encoding than the registered broker key,
triggering a bogus reassignment on every read/write cycle. Added a
findActiveBroker helper that falls back to an Equals-based scan and
canonicalizes the assignment in place so later writes are stable.
- broker_grpc_lookup.go isLockOwner: used raw string equality between
LockOwner() and BrokerAddress().String(), so a lock owner could fail
to recognize itself and proxy local lookup/config/admin RPCs away.
- pub_client/scheduler.go onEachAssignments: reused publisher jobs only
on exact LeaderBroker match, so an encoding flip in lookup results
tore down and recreated a stream to the same broker.
* fix(s3): honor ChecksumAlgorithm on presigned URL uploads
AWS SDK presigners hoist x-amz-sdk-checksum-algorithm (and related
checksum headers) into the signed URL's query string, so servers must
read either location. detectRequestedChecksumAlgorithm only looked at
request headers, so presigned PUTs with ChecksumAlgorithm set validated
and stored no additional checksum, and HEAD/GET never returned the
x-amz-checksum-* header.
Read these parameters from headers first, then fall back to a
case-insensitive query-string lookup. Apply the same fallback when
comparing an object-level checksum value against the computed one.
Fixes#9075
* test(s3): presigned URL checksum integration tests (#9075)
Adds test/s3/checksum with end-to-end coverage for flexible-checksum
behavior on presigned URL uploads. Tests generate a presigned PUT URL
with ChecksumAlgorithm set, upload the body with a plain http.Client
(bypassing AWS SDK middleware so the server must honor the query-string
hoisted x-amz-sdk-checksum-algorithm), then HEAD/GET with
ChecksumMode=ENABLED and assert the stored x-amz-checksum-* header.
Covers SHA256, SHA1, and a negative control with no checksum requested.
Wires the new directory into s3-go-tests.yml as its own CI job.
* perf(s3): parse presigned query once in detectRequestedChecksumAlgorithm
Previously, each header fallback called getHeaderOrQuery, which re-parsed
r.URL.Query() and allocated a new map on every invocation — up to eight
times per PutObject request. Parse the raw query at most once per request
(only when non-empty) and pass the pre-parsed url.Values into a new
lookupHeaderOrQuery helper.
Also drops a redundant strings.ToLower allocation in the case-insensitive
query key scan (strings.EqualFold already handles ASCII case folding).
Addresses review feedback from gemini-code-assist on PR #9076.
* test(s3): honor credential env vars and add presigned upload timeout
- init() now reads S3_ACCESS_KEY/S3_SECRET_KEY (and AWS_ACCESS_KEY_ID /
AWS_SECRET_ACCESS_KEY / AWS_REGION fallbacks) so that
`make test-with-server ACCESS_KEY=... SECRET_KEY=...` no longer
authenticates with hardcoded defaults while the server has been
started with different credentials.
- uploadViaPresignedURL uses a dedicated http.Client with a 30s timeout
instead of http.DefaultClient, so a stalled server fails fast in CI
instead of blocking until the suite's global timeout fires.
Addresses review feedback from coderabbitai on PR #9076.
* test(s3): pass S3_PORT and credentials through to checksum tests
- 'make test' now exports S3_ENDPOINT, S3_ACCESS_KEY, and S3_SECRET_KEY
derived from the Makefile variables so the Go test process talks to
the same endpoint/credentials that start-server was launched with.
- start-server cleans up the background SeaweedFS process and PID file
when the readiness poll times out, preventing stale port conflicts on
subsequent runs.
Addresses review feedback from coderabbitai on PR #9076.
* ci(s3): raise checksum tests step timeout
make test-with-server builds weed_binary, waits up to 90s for readiness,
then runs go test -timeout=10m. The previous 12-minute step timeout only
had ~2 minutes of headroom over the Go timeout, risking the Actions
runner killing the step before tests reported a real failure.
Bumps the job timeout from 15 to 20 minutes and the step timeout from
12 to 16 minutes, matching other S3 integration jobs.
Addresses review feedback from coderabbitai on PR #9076.
* perf(s3): thread pre-parsed query through putToFiler hot path
Parse the request's query string once at the top of putToFiler and
reuse the resulting url.Values for both the checksum-algorithm detection
and the expected-checksum verification. Previously, the verification
path called getHeaderOrQuery which re-parsed r.URL.Query() again on
every PutObject, defeating the previous commit's single-parse goal.
- Add parseRequestQuery + detectRequestedChecksumAlgorithmQ (the
pre-parsed-query variant). detectRequestedChecksumAlgorithm is now a
thin wrapper used by callers that do a single lookup.
- putToFiler parses once and threads the result through both call sites.
- Remove getHeaderOrQuery and update the unit test to use
lookupHeaderOrQuery directly.
Addresses follow-up review from gemini-code-assist on PR #9076.
* test(s3): check io.ReadAll error in uploadViaPresignedURL helper
* test(s3): drop SHA1 presigned test case
The AWS SDK v2 presigner signs a Content-MD5 header at presign time
for SHA1 PutObject requests even when no body is attached (the MD5 of
the empty payload gets baked into the signed headers). Uploading the
real body via a plain http.Client then trips SeaweedFS's MD5 validation
and returns BadDigest — an SDK/presigner quirk, not a SeaweedFS bug.
The SHA256 positive case already exercises the server-side
query-hoisted algorithm path that issue #9075 is about, and the unit
tests in weed/s3api cover each algorithm's header mapping. Drop the
SHA1 integration case rather than chase SDK-specific workarounds.
* test(s3): provide real Content-MD5 to presigned checksum test
AWS SDK v2's flexible-checksum middleware signs a Content-MD5 header at
presign time. There is no body to hash at that point, so it seeds the
header with MD5 of the empty payload. When the real body is then PUT
with a plain http.Client, SeaweedFS's server-side Content-MD5
verification correctly rejects the upload with BadDigest.
Pre-compute the MD5 of the test body and thread it into
PutObjectInput.ContentMD5 so the signed Content-MD5 matches the body
that will actually be uploaded. The test still exercises the
server-side path that reads X-Amz-Sdk-Checksum-Algorithm from the
query string (the fix that PR #9076 is validating).
* test(s3): send the signed Content-MD5 header on presigned upload
uploadViaPresignedURL now accepts an extraHeaders map so callers can
thread through headers that the presigner signed but the raw http
request would otherwise omit. The SHA256 test passes the Content-MD5
it computed, matching what the presigner baked into the signature.
Fixes SignatureDoesNotMatch seen in CI after the previous commit set
ContentMD5 on the presign input without sending the corresponding
header on the actual upload.
* test(s3): build presigned URL with the raw v4 signer
The AWS SDK v2 s3.PresignClient runs the flexible-checksum middleware
for any PutObject input that carries ChecksumAlgorithm. That middleware
injects a Content-MD5 header at presign time, and with no body present
it seeds MD5-of-empty. Any subsequent upload of a non-empty body
through a plain http.Client then trips SeaweedFS's Content-MD5
verification and returns BadDigest — not the code path that issue
#9075 is about.
Replace the PresignClient usage in the integration test with a direct
call to v4.Signer.PresignHTTP, building a canonical URL whose query
string already contains x-amz-sdk-checksum-algorithm=SHA256. This is
exactly the shape of URL a browser/curl client would receive from any
presigner that hoists the algorithm header, and it exercises the
server-side fix from PR #9076 without dragging in SDK-specific
middleware quirks.
* test(s3): set X-Amz-Expires on presigned URL before signing
v4.Signer.PresignHTTP does not add X-Amz-Expires on its own — the
caller has to seed it into the request's query string so the signer
includes it in the canonical query and the server accepts the
presigned URL. Without it, SeaweedFS correctly returns
AuthorizationQueryParametersError.
Also adds a .gitignore for the make-managed test volume data, log
file, and PID file so local `make test-with-server` runs do not leave
artifacts tracked by git.
Verified by running the integration tests locally:
make test-with-server → both presigned checksum tests PASS.
* add filer inode foundation for nfs
* nfs command skeleton
* add filer inode index foundation for nfs
* make nfs inode index hardlink aware
* add nfs filehandle and inode lookup plumbing
* add read-only nfs frontend foundation
* add nfs namespace mutation support
* add chunk-backed nfs write path
* add nfs protocol integration tests
* add stale handle nfs coverage
* complete nfs hardlink and failover coverage
* add nfs export access controls
* add nfs metadata cache invalidation
* fix nfs chunk read lookup routing
* fix nfs review findings and rename regression
* address pr 9067 review comments
- filer_inode: fail fast if the snowflake sequencer cannot start, and let
operators override the 10-bit node id via SEAWEEDFS_FILER_SNOWFLAKE_ID
to avoid multi-filer collisions
- filer_inode: drop the redundant retry loop in nextInode
- filerstore_wrapper: treat inode-index writes/removals as best-effort so
a primary store success no longer surfaces as an operation failure
- filer_grpc_server_rename: defer overwritten-target chunk deletion until
after CommitTransaction so a rolled-back rename does not strand live
metadata pointing at freshly deleted chunks
- command/nfs: default ip.bind to loopback and require an explicit
filer.path, so the experimental server does not expose the entire
filer namespace on first run
- nfs integration_test: document why LinkArgs matches go-nfs's on-the-wire
layout rather than RFC 1813 LINK3args
* mount: pre-allocate inode in Mkdir and Symlink
Mkdir and Symlink used to send filer_pb.CreateEntryRequest with
Attributes.Inode = 0. After PR 9067, the filer's CreateEntry now assigns
its own inode in that case, so the filer-side entry ends up with a
different inode than the one the mount allocates via inodeToPath.Lookup
and returns to the kernel. Once applyLocalMetadataEvent stores the
filer's entry in the meta cache, subsequent GetAttr calls read the
cached entry and hit the setAttrByPbEntry override at line 197 of
weedfs_attr.go, returning the filer-assigned inode instead of the
mount's local one. pjdfstest tests/rename/00.t (subtests 81/87/91)
caught this — it lstat'd a freshly-created directory/symlink, renamed
it, lstat'd again, and saw a different inode the second time.
createRegularFile already pre-allocates via inodeToPath.AllocateInode
and stamps it into the create request. Do the same thing in Mkdir and
Symlink so both sides agree on the object identity from the very first
request, and so GetAttr's cache path returns the same value as Mkdir /
Symlink's initial response.
* sequence: mask snowflake node id on int→uint32 conversion
CodeQL flagged the unchecked uint32(snowflakeId) cast in
NewSnowflakeSequencer as a potential truncation bug when snowflakeId is
sourced from user input (e.g. via SEAWEEDFS_FILER_SNOWFLAKE_ID). Mask
to the 10 bits the snowflake library actually uses so any caller-
supplied int is safely clamped into range.
* add test/nfs integration suite
Boots a real SeaweedFS cluster (master + volume + filer) plus the
experimental `weed nfs` frontend as subprocesses and drives it through
the NFSv3 wire protocol via go-nfs-client, mirroring the layout of
test/sftp. The tests run without a kernel NFS mount, privileged ports,
or any platform-specific tooling.
Coverage includes read/write round-trip, mkdir/rmdir, nested
directories, rename content preservation, overwrite + explicit
truncate, 3 MiB binary file, all-byte binary and empty files, symlink
round-trip, ReadDirPlus listing, missing-path remove, FSInfo sanity,
sequential appends, and readdir-after-remove.
Framework notes:
- Picks ephemeral ports with net.Listen("127.0.0.1:0") and passes
-port.grpc explicitly so the default port+10000 convention cannot
overflow uint16 on macOS.
- Pre-creates the /nfs_export directory via the filer HTTP API before
starting the NFS server — the NFS server's ensureIndexedEntry check
requires the export root to exist with a real entry, which filer.Root
does not satisfy when the export path is "/".
- Reuses the same rpc.Client for mount and target so go-nfs-client does
not try to re-dial via portmapper (which concatenates ":111" onto the
address).
* ci: add NFS integration test workflow
Mirror test/sftp's workflow for the new test/nfs suite so PRs that touch
the NFS server, the inode filer plumbing it depends on, or the test
harness itself run the 14 NFSv3-over-RPC integration tests on Ubuntu
22.04 via `make test`.
* nfs: use append for buffer growth in Write and Truncate
The previous make+copy pattern reallocated the full buffer on every
extending write or truncate, giving O(N^2) behaviour for sequential
write loops. Switching to `append(f.content, make([]byte, delta)...)`
lets Go's amortized growth strategy absorb the repeated extensions.
Called out by gemini-code-assist on PR 9067.
* filer: honor caller cancellation in collectInodeIndexEntries
Dropping the WithoutCancel wrapper lets DeleteFolderChildren bail out of
the inode-index scan if the client disconnects mid-walk. The cleanup is
already treated as best-effort by the caller (it logs on error and
continues), so a cancelled walk just means the partial index rebuild is
skipped — the same failure mode as any other index write error.
Flagged as a DoS concern by gemini-code-assist on PR 9067.
* nfs: skip filer read on open when O_TRUNC is set
openFile used to unconditionally loadWritableContent for every writable
open and then discard the buffer if O_TRUNC was set. For large files
that is a pointless 64 MiB round-trip. Reorder the branches so we only
fetch existing content when the caller intends to keep it, and mark the
file dirty right away so the subsequent Close still issues the
truncating write. Called out by gemini-code-assist on PR 9067.
* nfs: allow Seek on O_APPEND files and document buffered write cap
Two related cleanups on filesystem.go:
- POSIX only restricts Write on an O_APPEND fd, not lseek. The existing
Seek error ("append-only file descriptors may only seek to EOF")
prevented read-and-write workloads that legitimately reposition the
read cursor. Write already snaps the offset to EOF before persisting
(see seaweedFile Write), so Seek can unconditionally accept any
offset. Update the unit test that was asserting the old behaviour.
- Add a doc comment on maxBufferedWriteSize explaining that it is a
per-file ceiling, the memory footprint it implies, and that the real
fix for larger whole-file rewrites is streaming / multi-chunk support.
Both changes flagged by gemini-code-assist on PR 9067.
* nfs: guard offset before casting to int in Write
CodeQL flagged `int(f.offset) + len(p)` inside the Write growth path as
a potential overflow on architectures where `int` is 32-bit. The
existing check only bounded the post-cast value, which is too late.
Clamp f.offset against maxBufferedWriteSize before the cast and also
reject negative/overflowed endOffset results. Both branches fall
through to billy.ErrNotSupported, the same behaviour the caller gets
today for any out-of-range buffered write.
* nfs: compute Write endOffset in int64 to satisfy CodeQL
The previous guard bounded f.offset but left len(p) unchecked, so
CodeQL still flagged `int(f.offset) + len(p)` as a possible int-width
overflow path. Bound len(p) against maxBufferedWriteSize first, do the
addition in int64, and only cast down after the total has been clamped
against the buffer ceiling. Behaviour is unchanged: any out-of-range
write still returns billy.ErrNotSupported.
* ci: drop emojis from nfs-tests workflow summary
Plain-text step summary per user preference — no decorative glyphs in
the NFS CI output or checklist.
* nfs: annotate remaining DEV_PLAN TODOs with status
Three of the unchecked items are genuine follow-up PRs rather than
missing work in this one, and one was actually already done:
- Reuse chunk cache and mutation stream helpers without FUSE deps:
checked off — the NFS server imports weed/filer.ReaderCache and
weed/util/chunk_cache directly with no weed/mount or go-fuse imports.
- Extract shared read/write helpers from mount/WebDAV/SFTP: annotated
as deferred to a separate refactor PR (touches four packages).
- Expand direct data-path writes beyond the 64 MiB buffered fallback:
annotated as deferred — requires a streaming WRITE path.
- Shared lock state + lock tests: annotated as blocked upstream on
go-nfs's missing NLM/NFSv4 lock state RPCs, matching the existing
"Current Blockers" note.
* test/nfs: share port+readiness helpers with test/testutil
Drop the per-suite mustPickFreePort and waitForService re-implementations
in favor of testutil.MustAllocatePorts (atomic batch allocation; no
close-then-hope race) and testutil.WaitForPort / SeaweedMiniStartupTimeout.
Pull testutil in via a local replace directive so this standalone
seaweedfs-nfs-tests module can import the in-repo package without a
separate release.
Subprocess startup is still master + volume + filer + nfs — no switch to
weed mini yet, since mini does not know about the nfs frontend.
* nfs: stream writes to volume servers instead of buffering the whole file
Before this change the NFS write path held the full contents of every
writable open in memory:
- OpenFile(write) called loadWritableContent which read the existing
file into seaweedFile.content up to maxBufferedWriteSize (64 MiB)
- each Write() extended content in-place
- Close() uploaded the whole buffer as a single chunk via
persistContent + AssignVolume
The 64 MiB ceiling made large NFS writes return NFS3ERR_NOTSUPP, and
even below the cap every Write paid a whole-file-in-memory cost. This
PR rewrites the write path to match how `weed filer` and the S3 gateway
persist data:
- openFile(write) no longer loads the existing content at all; it
only issues an UpdateEntry when O_TRUNC is set *and* the file is
non-empty (so a fresh create+trunc is still zero-RPC)
- Write() streams the caller's bytes straight to a volume server via
one AssignVolume + one chunk upload, then atomically appends the
resulting chunk to the filer entry through mutateEntry. Any
previously inlined entry.Content is migrated to a chunk in the same
update so the chunk list becomes the authoritative representation.
- Truncate() becomes a direct mutateEntry (drop chunks past the new
size, clip inline content, update FileSize) instead of resizing an
in-memory buffer.
- Close() is a no-op because everything was flushed inline.
The small-file fast path that the filer HTTP handler uses is preserved:
if the post-write size still fits in maxInlineWriteSize (4 MiB) and
the file has no existing chunks, we rewrite entry.Content directly and
skip the volume-server round-trip. This keeps single-shot tiny writes
(echo, small edits) cheap while completely removing the 64 MiB cap on
larger files. Read() now always reads through the chunk reader instead
of a local byte slice, so reads inside the same session see the freshly
appended data.
Drops the unused seaweedFile.content / dirty fields, the
maxBufferedWriteSize constant, and the loadWritableContent helper.
Updates TestSeaweedFileSystemSupportsNamespaceMutations expectations
to match the new "no extra O_TRUNC UpdateEntry on an empty file"
behavior (still 3 updates: Write + Chmod + Truncate).
* filer: extract shared gateway upload helper for NFS and WebDAV
Three filer-backed gateways (NFS, WebDAV, and mount) each had a local
saveDataAsChunk that wrapped operation.NewUploader().UploadWithRetry
with near-identical bodies: build AssignVolumeRequest, build
UploadOption, build genFileUrlFn with optional filerProxy rewriting,
call UploadWithRetry, validate the result, and call ToPbFileChunk.
Pull that body into filer.SaveGatewayDataAsChunk with a
GatewayChunkUploadRequest struct so both NFS and WebDAV can delegate
to one implementation.
- NFS's saveDataAsChunk is now a thin adapter that assembles the
GatewayChunkUploadRequest from server options and calls the helper.
The chunkUploader interface keeps working for test injection because
the new GatewayChunkUploader interface is structurally identical.
- WebDAV's saveDataAsChunk is similarly a thin adapter — it drops the
local operation.NewUploader call plus the AssignVolume/UploadOption
scaffolding.
- mount is intentionally left alone. mount's saveDataAsChunk has two
features that do not fit the shared helper (a pre-allocated file-id
pool used to skip AssignVolume entirely, and a chunkCache
write-through at offset 0 so future reads hit the mount's local
cache), both of which are mount-specific.
Marks the Phase 2 "extract shared read/write helpers from mount,
WebDAV, and SFTP" DEV_PLAN item as done. The filer-level chunk read
path (NonOverlappingVisibleIntervals + ViewFromVisibleIntervals +
NewChunkReaderAtFromClient) was already shared.
* nfs: remove DESIGN.md and DEV_PLAN.md
The planning documents have served their purpose — all phase 1 and
phase 2 items are landed, phase 3 streaming writes are landed, phase 2
shared helpers are extracted, and the two remaining phase 4 items
(shared lock state + lock tests) are blocked upstream on
github.com/willscott/go-nfs which exposes no NLM or NFSv4 lock state
RPCs. The running decision log no longer reflects current code and
would just drift. The NFS wiki page
(https://github.com/seaweedfs/seaweedfs/wiki/NFS-Server) now carries
the overview, configuration surface, architecture notes, and known
limitations; the source is the source of truth for the rest.
* fix(iceberg): clean stale data before creating a table
CREATE TABLE AS from Trino fails against the SeaweedFS Iceberg REST
catalog with "Cannot create a table on a non-empty location". The
catalog assigns every new table the deterministic <ns>/<table> path,
and Trino's pre-write check rejects the CTAS whenever leftover objects
live there — typically files from a prior DROP that did not purge the
data, or an earlier aborted CTAS.
Make the catalog the authority for table existence: before writing any
metadata, look up the table in the S3 Tables catalog. If it is already
registered, return the existing definition (idempotent create). If it
is not registered, any objects still sitting at the target location
are stale, so purge them before proceeding. Live tables are never
touched — the cleanup path is guarded by the catalog lookup.
Fixes#9074
* test(trino): regression for create/drop/recreate table (#9074)
Exercises the exact sequence from the reported bug: CREATE TABLE without
an explicit location, INSERT, DROP, then CREATE again with the same name,
followed by a CTAS on top. Previously the recreate failed with
"Cannot create a table on a non-empty location" because stale data files
from the dropped table lingered at the deterministic <schema>/<table>
path. The test also asserts the recreated table does not see the dropped
data and that a drop/recreate CTAS cycle works.
* fix(iceberg): purge dropped table location on DROP TABLE
Recreate-at-same-location was failing with "Cannot create a table on a
non-empty location" because DROP TABLE only removed the catalog entry
and left data files behind. Trino's pre-write emptiness check then
rejected the subsequent CREATE.
Look up the table's storage location before deleting the catalog entry,
and after a successful DeleteTable, purge the filer subtree at that
location. The lookup uses the catalog — the authoritative owner of the
name→location mapping — so cleanup is gated on a live table having
existed; failed lookups skip cleanup and leave storage alone.
Also pin the regression test to an explicit, fixed location so the
recreate genuinely targets the same path the drop was supposed to free.
* refactor(iceberg): use errors.As in isNoSuchTableError
* fix(iceberg): drop create-preflight cleanup; harden cleanup path
- Remove the destructive cleanupStaleTableLocation call from the
CreateTable preflight. Storage cleanup now lives exclusively on the
DROP path, so CreateTable has no side effects on storage beyond the
new table's own metadata write.
- Validate tablePath in cleanupStaleTableLocation by rejecting empty,
".", "..", or backslash-bearing segments before joining with the
bucket prefix, so a crafted location cannot escape the bucket
subtree. path.Clean would silently collapse traversal, so segments
are checked raw.
* test(trino): defer table drops in recreate test for failure-safe cleanup
* fix(helm): skip s3 ServiceMonitor when only filer.s3 is enabled (#9080)
The seaweedfs-s3 Service only exposes a "metrics" port when the standalone
s3 gateway is enabled. With filer.s3.enabled=true and s3.enabled=false the
Service only has swfs-s3:8333, so the generated ServiceMonitor matched zero
targets and fired persistent no-targets alerts. The embedded filer S3
gateway's metrics are already scraped via the filer ServiceMonitor.
* comment: drop issue ref
* perf(filer.sync): don't serialize descendants behind dir attribute updates
The MetadataProcessor treated every in-flight directory job as a subtree
barrier: any active dir job at /foo forced all file events under /foo to
wait, and because the admit loop runs on the single stream.Recv()
goroutine, a stalled descendant also stalled the whole gRPC stream. For
large directories this turned every attribute-only dir event (mtime /
xattr / chmod bumps) into a full-subtree pinch point.
Classify dir jobs as barrier (create / delete / rename) vs non-barrier
(filer_pb.IsUpdate on a directory — same parent and same name, i.e. an
in-place attribute update). Only barrier dirs block descendants and get
blocked by ancestor barrier dirs. Non-barrier dir updates still bump the
ancestor descendantCount, so an incoming barrier dir on an ancestor
still waits for them — preserving the "delete /a waits for in-flight
/a/b update" safety.
Tests cover the loosened cases and the preserved barriers:
non-barrier update doesn't block a file descendant, barrier create
still does, barrier delete still waits for in-flight descendants, and
a barrier ancestor still waits for a non-barrier descendant update.
* fix(filer.sync): serialize same-path barrier dir jobs against concurrent ops
Review (Gemini) flagged that pathConflicts had latent same-path gaps
that predated this PR but deserve fixing alongside the dir-conflict
loosening: two barrier dir jobs at the same path could run concurrently
(e.g. create /a and delete /a), and a file job at the same path as an
in-flight barrier dir wasn't blocked either.
Tighten pathConflicts so that:
- an active barrier dir at p blocks every incoming job at p (file,
barrier dir, or non-barrier attribute update) — same-path promotions,
renames, and delete/create collisions must serialize;
- an active file at p blocks incoming files and barrier dirs at p;
- non-barrier dir updates at the same path still overlap with each
other (attribute bumps are last-writer-wins, intentional).
TestDirVsDirConflict and TestFileUnderActiveDirConflict flip their
"same path does not conflict" assertions to match. New
TestSamePathBarrierSerialization covers all five same-path cases
explicitly.
* fix(filer.sync): serialize incoming barrier dir against same-path non-barrier update
Bug introduced by the previous same-path tightening commit and caught
in review (CodeRabbit, critical): a kindNonBarrierDir at /dir1 was not
indexed at its own path, so a later kindBarrierDir at /dir1 saw neither
activeBarrierDirPaths["/dir1"] nor descendantCount["/dir1"] (the latter
only counts strict descendants) and was admitted concurrently with the
in-flight attribute update. That violated the "barrier at p serializes
all work at p" rule.
Track non-barrier dir jobs in a new activeNonBarrierDirPaths map and
check it only from the incoming-barrier-dir branch of pathConflicts.
The map is deliberately invisible to the ancestor check, so non-barrier
updates still don't serialize file descendants — the loosening this PR
is about stays intact.
Regression test added in TestSamePathBarrierSerialization covers both
the admission conflict and the index cleanup on job completion.
* fix(s3): allow anonymous ListBuckets with prefix-scoped List action
An anonymous identity holding a prefix-scoped action such as
"List:prefix-*" was denied at the auth middleware before ListBucketsHandler
could apply the per-bucket visibility check. The middleware called
CanDo with an empty bucket, which never matches a scoped action, so
every anonymous ListBuckets request returned 403 even though matching
buckets should have been visible.
Defer ListBuckets authorization to the handler for the anonymous
identity when it actually carries a List action, mirroring the
existing behavior for authenticated users. Anonymous identities with
no List action continue to be rejected at the global layer, preserving
the secure-by-default posture.
Fixes#9072
* refactor(s3): make hasListAction a method on Identity
Addresses PR review — consistent with existing CanDo/isAdmin methods and
also treats Admin identities as implicitly having List permission.
Adds the -s3.cacheCapacityMB flag (default 0, disabled) that attaches
an in-memory chunk_cache.ChunkCacheInMemory to the server-wide
ReaderCache introduced in the previous commit. When enabled,
completed chunks are deposited into the shared cache as they are
downloaded, so concurrent and repeat GETs of the same object hit
memory instead of re-fetching chunks from volume servers.
When 0 (the default) the shared ReaderCache still runs — it just
attaches a nil chunk cache, so behaviour matches the previous commit
exactly. No behaviour change for clusters that don't opt in.
Disk-backed TieredChunkCache was evaluated and rejected: its
synchronous SetChunk writes regressed cold reads ~12x on loopback
because the chunk fetchers block on local disk I/O that is *slower*
than the TCP volume-server fetch it is supposed to accelerate.
Memory-only avoids that.
Flag registered in all four S3 flag sites (s3.go, server.go,
filer.go, mini.go) per the comment on command.S3Options. The chunk
size used to convert CacheSizeMB → entry count is encapsulated in
the s3ChunkCacheChunkSizeMB constant so it's easy to grep and
revisit if the filer default chunk size changes.
Measured on weed mini + 1 GiB random object over loopback, single
curl on a presigned URL:
cacheCapacityMB=0 (off): cold ~2900, warm ~2900 MB/s
cacheCapacityMB=4096: cold ~2790, warm ~5050 MB/s (+70%)
* feat(mount): cap write buffer with -writeBufferSizeMB
Without a bound on the per-mount write pipeline, sustained upload
failures (e.g. volume server returning "Volume Size Exceeded" while
the master hasn't yet rotated assignments) let sealed chunks pile up
across open file handles until the swap directory — by default
os.TempDir() — fills the disk. Reported on 4.19 filling /tmp to 1.8 TB
during a large rclone sync.
Add a global WriteBufferAccountant shared across every UploadPipeline
in a mount. Creating a new page chunk (memory or swap) first reserves
ChunkSize bytes; when the cap is reached the writer blocks until an
uploader finishes and releases, turning swap overflow into natural
FUSE-level backpressure instead of unbounded disk growth.
The new -writeBufferSizeMB flag (also accepted via fuse.conf) defaults
to 0 = unlimited, preserving current behavior. Reserve drops
chunksLock while blocking so uploader goroutines — which take
chunksLock on completion before calling Release — cannot deadlock,
and an oversized reservation on an empty accountant succeeds to avoid
single-handle starvation.
* fix(mount): plug write-budget leaks in pipeline Shutdown
Review on #9066 caught two accounting bugs on the Destroy() path:
1. Writable-chunk leak (high). SaveDataAt() reserves ChunkSize before
inserting into writableChunks, but Shutdown() only iterated
sealedChunks. Truncate / metadata-invalidation flows call Destroy()
(via ResetDirtyPages) without flushing first, so any dirty but
unsealed chunks would permanently shrink the global write budget.
Shutdown now frees and releases writable chunks too.
2. Double release with racing uploader (medium). Shutdown called
accountant.Release directly after FreeReference, while the async
uploader goroutine did the same on normal completion — under a
Destroy-before-flush race this could underflow the accountant and
let later writes exceed the configured cap. Move accounting into
SealedChunk.FreeReference itself: the refcount-zero transition is
exactly-once by construction, so any number of FreeReference calls
release the slot precisely once.
Add regression tests for the writable-leak and the FreeReference
idempotency guarantee.
* test(mount): remove sleep-based race in accountant blocking test
Address review nits on #9066:
- Replace time.Sleep(50ms) proxy for "goroutine entered Reserve" with
a started channel the goroutine closes immediately before calling
Reserve. Reserve cannot make progress until Release is called, so
landed is guaranteed false after the handshake — no arbitrary wait.
- Short-circuit WriteBufferAccountant.Used() in unlimited mode for
consistency with Reserve/Release, avoiding a mutex round-trip.
* test(mount): add end-to-end write-buffer cap integration test
Exercises the full write-budget plumbing with a small cap (4 chunks of
64 KiB = 256 KiB) shared across three UploadPipelines fed by six
concurrent writers. A gated saveFn models the "volume server rejecting
uploads" condition from the original report: no sealed chunk can drain
until the test opens the gate. A background sampler records the peak
value of accountant.Used() throughout the run.
The test asserts:
- writers fill the budget and then block on Reserve (Used() stays at
the cap while stalled)
- Used() never exceeds the configured cap even under concurrent
pressure from multiple pipelines
- after the gate opens, writers drain to zero
- peak observed Used() matches the cap (262144 bytes in this run)
While wiring this up, the race detector surfaced a pre-existing data
race on UploadPipeline.uploaderCount: the two glog.V(4) lines around
the atomic Add sites read the field non-atomically. Capture the new
value from AddInt32 and log that instead — one-liner each, no
behavioral change.
* test(fuse): end-to-end integration test for -writeBufferSizeMB
Exercise the new write-buffer cap against a real weed mount so CI
(fuse-integration.yml) covers the FUSE→upload-pipeline→filer path, not
just the in-package unit tests. Uses a 4 MiB cap with 2 MiB chunks so
every subtest's total write demand is multiples of the budget and
Reserve/Release must drive forward progress for writes to complete.
Subtests:
- ConcurrentLargeWrites: six parallel 6 MiB files (36 MiB total, ~18
chunk allocations) through the same mount, verifies every byte
round-trips.
- SingleFileExceedingCap: one 20 MiB file (10 chunks) through a single
handle, catching any self-deadlock when the pipeline's own earlier
chunks already fill the global budget.
- DoesNotDeadlockAfterPressure: final small write with a 30s timeout,
catching budget-slot leaks that would otherwise hang subsequent
writes on a still-full accountant.
Ran locally on Darwin with macfuse against a real weed mini + mount:
=== RUN TestWriteBufferCap
--- PASS: TestWriteBufferCap (1.82s)
* test(fuse): loosen write-buffer cap e2e test + fail-fast on hang
On Linux CI the previous configuration (-writeBufferSizeMB=4,
-concurrentWriters=4 against a 20 MiB single-handle write)
deterministically hung the "Run FUSE Integration Tests" step to the
45-minute workflow timeout, while on macOS / macfuse the same test
completes in ~2 seconds (see run 24386197483). The Linux hang shows
up after TestWriteBufferCap/ConcurrentLargeWrites completes cleanly,
then TestWriteBufferCap/SingleFileExceedingCap starts and never
emits its PASS line.
Change:
- Loosen the cap to 16 MiB (8 × 2 MiB chunk slots) and drop the
custom -concurrentWriters override. The subtests still drive demand
well above the cap (32 MiB concurrent, 12 MiB single-handle), so
Reserve/Release is still on every chunk-allocation path; the cap
just gives the pipeline enough headroom that interactions with the
per-file writableChunkLimit and the go-fuse MaxWrite batching don't
wedge a single-handle writer on a slow runner.
- Wrap every os.WriteFile in a writeWithTimeout helper that dumps every
live goroutine on timeout. If this ever re-regresses, CI surfaces
the actual stuck goroutines instead of a 45-minute walltime.
- Also guard the concurrent-writer goroutines with the same timeout +
stack dump.
The in-package unit test TestWriteBufferCap_SharedAcrossPipelines
remains the deterministic, controlled verification of the blocking
Reserve/Release path — this e2e test is now a smoke test for
correctness and absence of deadlocks through a real FUSE mount, which
is all it should be.
* fix: address PR #9066 review — idempotent FreeReference, subtest watchdog, larger single-handle test
FreeReference on SealedChunk now early-returns when referenceCounter is
already <= 0. The existing == 0 body guard already made side effects
idempotent, but the counter itself would still decrement into the
negatives on a double-call — ugly and a latent landmine for any future
caller that does math on the counter. Make double-call a strict no-op.
test(fuse): per-subtest watchdog + larger single-handle test
- Add runSubtestWithWatchdog and wrap every TestWriteBufferCap subtest
with a 3-minute deadline. Individual writes were already
timeout-wrapped but the readback loops and surrounding bookkeeping
were not, leaving a gap where a subtest body could still hang. On
watchdog fire, every live goroutine is dumped so CI surfaces the
wedge instead of a 45-minute walltime.
- Bump testLargeFileUnderCap from 12 MiB → 20 MiB (10 chunks) to
exceed the 16 MiB cap (8 slots) again and actually exercise
Reserve/Release backpressure on a single file handle. The earlier
e2e hang was under much tighter params (-writeBufferSizeMB=4,
-concurrentWriters=4, writable limit 4); with the current loosened
config the pressure is gentle and the goroutine-dump-on-timeout
safety net is in place if it ever regresses.
Declined: adding an observable peak-Used() assertion to the e2e test.
The mount runs as a subprocess so its in-process WriteBufferAccountant
state isn't reachable from the test without adding a metrics/RPC
surface. The deterministic peak-vs-cap verification already lives in
the in-package unit test TestWriteBufferCap_SharedAcrossPipelines.
Recorded this rationale inline in TestWriteBufferCap's doc comment.
* test(fuse): capture mount pprof goroutine dump on write-timeout
The previous run (24388549058) hung on LargeFileUnderCap and the
test-side dumpAllGoroutines only showed the test process — the test's
syscall.Write is blocked in the kernel waiting for FUSE to respond,
which tells us nothing about where the MOUNT is stuck. The mount runs
as a subprocess so its in-process stacks aren't reachable from the
test.
Enable the mount's pprof endpoint via -debug=true -debug.port=<free>,
allocate the port from the test, and on write-timeout fetch
/debug/pprof/goroutine?debug=2 from the mount process and log it. This
gives CI the only view that can actually diagnose a write-buffer
backpressure deadlock (writer goroutines blocked on Reserve, uploader
goroutines stalled on something, etc).
Kept fileSize at 20 MiB so the Linux CI run will still hit the hang
(if it's genuinely there) and produce an actionable mount-side dump;
the alternative — silently shrinking the test below the cap — would
lose the regression signal entirely.
* review: constructor-inject accountant + subtest watchdog body on main
Two PR-#9066 review fixes:
1. NewUploadPipeline now takes the WriteBufferAccountant as a
constructor parameter; SetWriteBufferAccountant is removed. In
practice the previous setter was only called once during
newMemoryChunkPages, before any goroutine could touch the
pipeline, so there was no actual race — but constructor injection
makes the "accountant is fixed at construction time" invariant
explicit and eliminates the possibility of a future caller
mutating it mid-flight. All three call sites (real + two tests)
updated; the legacy TestUploadPipeline passes a nil accountant,
preserving backward-compatible unlimited-mode behavior.
2. runSubtestWithWatchdog now runs body on the subtest main goroutine
and starts a watcher goroutine that only calls goroutine-safe t
methods (t.Log, t.Logf, t.Errorf). The previous version ran body
on a spawned goroutine, which meant any require.* or writeWithTimeout
t.Fatalf inside body was being called from a non-test goroutine —
explicitly disallowed by Go's testing docs. The watcher no longer
interrupts body (it can't), so body must return on its own —
which it does via writeWithTimeout's internal 90s timeout firing
t.Fatalf on (now) the main goroutine. The watchdog still provides
the critical diagnostic: on timeout it dumps both test-side and
mount-side (via pprof) goroutine stacks and marks the test failed
via t.Errorf.
* fix(mount): IsComplete must detect coverage across adjacent intervals
Linux FUSE caps per-op writes at FUSE_MAX_PAGES_PER_REQ (typically
1 MiB on x86_64) regardless of go-fuse's requested MaxWrite, so a
2 MiB chunk filled by a sequential writer arrives as two adjacent
1 MiB write ops. addInterval in ChunkWrittenIntervalList does not
merge adjacent intervals, so the resulting list has two elements
{[0,1M], [1M,2M]} — fully covered, but list.size()==2.
IsComplete previously returned `list.size() == 1 &&
list.head.next.isComplete(chunkSize)`, which required a single
interval covering [0, chunkSize). Under that rule, chunks filled by
adjacent writes never reach IsComplete==true, so maybeMoveToSealed
never fires, and the chunks sit in writableChunks until
FlushAll/close. SaveContent handles the adjacency correctly via its
inline merge loop, so uploads work once they're triggered — but
IsComplete is the gate that triggers them.
This was a latent bug: without the write-buffer cap, the overflow
path kicks in at writableChunkLimit (default 128) and force-seals
chunks, hiding the leak. #9066's -writeBufferSizeMB adds a tighter
global cap, and with 8 slots / 20 MiB test, the budget trips long
before overflow. The writer blocks in Reserve, waiting for a slot
that never frees because no uploader ever ran — observed in the CI
run 24390596623 mount pprof dump: goroutine 1 stuck in
WriteBufferAccountant.Reserve → cond.Wait, zero uploader goroutines
anywhere in the 89-goroutine dump.
Walk the (sorted) interval list tracking the furthest covered
offset; return true if coverage reaches chunkSize with no gaps. This
correctly handles adjacent intervals, overlapping intervals, and
out-of-order inserts. Added TestIsComplete_AdjacentIntervals
covering single-write, two adjacent halves (both orderings), eight
adjacent eighths, gaps, missing edges, and overlaps.
* test(fuse): route mount glog to stderr + dump mount on any write error
Run 24392087737 (with the IsComplete fix) no longer hangs on Linux —
huge progress. Now TestWriteBufferCap/LargeFileUnderCap fails with
'close(...write_buffer_cap_large.bin): input/output error', meaning
a chunk upload failed and pages.lastErr propagated via FlushData to
close(). But the mount log in the CI artifact is empty because weed
mount's glog defaults to /tmp/weed.* files, which the CI upload step
never sees, so we can't tell WHICH upload failed or WHY.
Add -logtostderr=true -v=2 to MountOptions so glog output goes to
the mount process's stderr, which the framework's startProcess
redirects into f.logDir/mount.log, which the framework's DumpLogs
then prints to the test output on failure. The -v=2 floor enables
saveDataAsChunk upload errors (currently logged at V(0)) plus the
medium-level write_pipeline/upload traces without drowning the log
in V(4) noise.
Also dump MOUNT goroutines on any writeWithTimeout error (not just
timeout). The IsComplete fix means we now get explicit errors
instead of silent hangs, and the goroutine dump at the error moment
shows in-flight upload state (pending sealed chunks, retry loops,
etc) that a post-failure log alone can't capture.
perf(s3): route GET through ChunkReadAt + shared ReaderCache
The S3 GET path previously used filer.PrepareStreamContentWithPrefetch,
which hands chunk bytes from the volume-server fetch goroutine to the
consumer through an io.Pipe. io.Pipe is a synchronous rendezvous, so
the prefetch=4 window only overlapped HTTP connection setup — the
actual data bytes still flowed one pipe at a time.
Switch to the same path WebDAV uses (server/webdav_server.go): build
a filer.ChunkReadAt backed by a server-wide filer.ReaderCache.
ReaderCache prefetches whole chunks into []byte buffers, so the
prefetch window translates into real in-flight bytes and the consumer
copies them out as memcpys.
The ReaderCache is server-wide (not per-request) for two reasons:
1. ChunkReadAt.Close() destroys the ReaderCache's downloader map.
With a per-request cache, the defer on the handler would wait for
background chunk downloads that run on context.Background() — so
a client disconnect would block handler cleanup on downloads that
the client no longer wants, tying up goroutines and memory.
2. Concurrent requests for the same object can share in-flight
downloads through the shared downloader map.
No persistent ChunkCache is added in this commit — the ReaderCache is
constructed with a nil *chunk_cache.TieredChunkCache (all its methods
are nil-receiver safe). A follow-up PR wires in an in-memory chunk
cache for cross-request warm hits.
JWT for volume-server requests is generated internally by
util_http.RetriedFetchChunkData from jwtSigningReadKey, so the new
path remains compatible with JWT-protected clusters — this is the
same mechanism the WebDAV and mount read paths have been using.
Measured on weed mini + 1 GiB random object over loopback, cold
cache, single-stream curl on a presigned URL:
before (io.Pipe): 2100-2200 MB/s
after (ChunkReadAt): 2900-3800 MB/s
Trigger the helm release workflow automatically on tag pushes so each
software release also publishes the chart to gh-pages and the OCI
registry at ghcr.io/seaweedfs. workflow_dispatch is kept as a manual
fallback.
Refs #6296
build(docker): apply full apk upgrade in final image to pick up security patches
Trivy flagged CVE-2026-28390 (libcrypto3/libssl3) on the published image
because the final stage only upgraded zlib. Broaden to `apk upgrade
--no-cache` so all Alpine security fixes land at build time.
* fix(mount): serialize hard-link mutations on HardLinkId
syncHardLinkSiblings stamps every sibling of a hard-link to
authoritativeEntry.HardLinkCounter, and the caller computes that value
as entry.HardLinkCounter - 1 (Unlink) or entry.HardLinkCounter + 1
(Link) from a cached entry read before the filer mutation. With
concurrent Unlinks on different links of the same file, both callers
observe the same pre-decrement counter, the filer's atomic blob
decrement lands correctly, but both then stamp their siblings to
counter-1 — leaving the mount metacache one higher than the authoritative
blob.
Serialize Link and Unlink on string(HardLinkId) via a new
hardLinkLockTable on WFS, and re-load the entry under the lock so the
second caller sees the updated sibling counter its predecessor just
wrote before computing its own delta. First-link races (empty
HardLinkId on the source) are a separate pre-existing issue and are
not addressed here.
Full pjdfstest suite still passes (235 files, 8803 tests).
* fix(mount): abort on stale pre-lock entry after HardLinkId lock
Review follow-up: if maybeLoadEntry fails after acquiring the
hardLinkLockTable lock, the prior revision silently fell back to the
pre-lock snapshot, reintroducing the stale-base update the lock is
meant to prevent.
- Unlink: treat fuse.ENOENT as success (the file was already removed by
the thread that held the lock before us) and propagate any other
error.
- Link: abort with the returned status so we never derive the next
HardLinkCounter from a stale source entry.
* fix(mount): re-resolve Link source alias under HardLinkId lock
Review follow-up: Link resolved oldEntryPath from in.Oldnodeid before
waiting on the HardLinkId lock. A concurrent Unlink that held the same
lock could remove the specific alias we picked pre-lock while leaving
other sibling hard links for the same inode intact. The post-lock
maybeLoadEntry then returned ENOENT even though the source inode was
still reachable.
Call GetPath(in.Oldnodeid) again under the lock to pick whichever
alias is still active, refresh oldParentPath, and only return ENOENT
if no sibling survived.
* admin: report file and delete counts for EC volumes
The admin bucket size fix (#9058) left object counts at zero for
EC-encoded data because VolumeEcShardInformationMessage carried no file
count. Billing/monitoring dashboards therefore still under-report
objects once a bucket is EC-encoded.
Thread file_count and delete_count end-to-end:
- Add file_count/delete_count to VolumeEcShardInformationMessage (proto
fields 8 and 9) and regenerate master_pb.
- Compute them lazily on volume servers by walking the .ecx index once
per EcVolume, cache on the struct, and keep the cache in sync inside
DeleteNeedleFromEcx (distinguishing live vs already-tombstoned
entries so idempotent deletes do not drift the counts).
- Populate the new proto fields from EcVolume.ToVolumeEcShardInformationMessage
and carry them through the master-side EcVolumeInfo / topology sync.
- Aggregate in admin collectCollectionStats, deduping per volume id:
every node holding shards of an EC volume reports the same counts, so
summing across nodes would otherwise multiply the object count by the
number of shard holders.
Regression tests cover the initial .ecx walk, live/tombstoned delete
bookkeeping (including idempotent and missing-key cases), and the admin
dedup path for an EC volume reported by multiple nodes.
* ec: include .ecj journal in EcVolume delete count
The initial delete count only reflected .ecx tombstones, missing any
needle that was journaled in .ecj but not yet folded into .ecx — e.g.
on partial recovery. Expand initCountsLocked to take the union of
.ecx tombstones and .ecj journal entries, deduped by needle id, so:
- an id that is both tombstoned in .ecx and listed in .ecj counts once
- a duplicate .ecj entry counts once
- an .ecj id with a live .ecx entry is counted as deleted (not live)
- an .ecj id with no matching .ecx entry is still counted
Covered by TestEcVolumeFileAndDeleteCountEcjUnion.
* ec: report delete count authoritatively and tombstone once per delete
Address two issues with the previous EcVolume file/delete count work:
1. The delete count was computed lazily on first heartbeat and mixed
in a .ecj-union fallback to "recover" partial state. That diverged
from how regular volumes report counts (always live from the needle
map) and had drift cases when .ecj got reconciled. Replace with an
eager walk of .ecx at NewEcVolume time, maintained incrementally on
every DeleteNeedleFromEcx call. Semantics now match needle_map_metric:
FileCount is the total number of needles ever recorded in .ecx
(live + tombstoned), DeleteCount is the tombstones — so live =
FileCount - DeleteCount. Drop the .ecj-union logic entirely.
2. A single EC needle delete fanned out to every node holding a replica
of the primary data shard and called DeleteNeedleFromEcx on each,
which inflated the per-volume delete total by the replica factor.
Rewrite doDeleteNeedleFromRemoteEcShardServers to try replicas in
order and stop at the first success (one tombstone per delete), and
only fall back to other shards when the primary shard has no home
(ErrEcShardMissing sentinel), not on transient RPC errors.
Admin aggregation now folds EC counts correctly: FileCount is deduped
per volume id (every shard holder has an identical .ecx) and DeleteCount
is summed across nodes (each delete tombstones exactly one node). Live
object count = deduped FileCount - summed DeleteCount.
Tests updated to match the new semantics:
- EC volume counts seed FileCount as total .ecx entries (live +
tombstoned), DeleteCount as tombstones.
- DeleteNeedleFromEcx keeps FileCount constant and increments
DeleteCount only on live->tombstone transitions.
- Admin dedup test uses distinct per-node delete counts (5 + 3 + 2)
to prove they're summed, while FileCount=100 is applied once.
* ec: test fixture uses real vid; admin warns on skewed ec counts
- writeFixture now builds the .ecx/.ecj/.ec00/.vif filenames from the
actual vid passed in, instead of hardcoding "_1". The existing tests
all use vid=1 so behaviour is unchanged, but the helper no longer
silently diverges from its documented parameter.
- collectCollectionStats logs a glog warning when an EC volume's summed
delete count exceeds its deduped file count, surfacing the anomaly
(stale heartbeat, counter drift, etc.) instead of silently dropping
the volume from the object count.
* ec: derive file/delete counts from .ecx/.ecj file sizes
seedCountsFromEcx walked the full .ecx index at volume load, which is
wasted work: .ecx has fixed-size entries (NeedleMapEntrySize) and .ecj
has fixed-size deletion records (NeedleIdSize), so both counts are pure
file-size arithmetic.
fileCount = ecxFileSize / NeedleMapEntrySize
deleteCount = ecjFileSize / NeedleIdSize
Rip out the cached counters, countsLock, seedCountsFromEcx, and the
recordDelete helper. Track ecjFileSize directly on the EcVolume struct,
seed it from Stat() at load, and bump it on every successful .ecj append
inside DeleteNeedleFromEcx under ecjFileAccessLock. Skip the .ecj write
entirely when the needle is already tombstoned so the derived delete
count stays idempotent on repeat deletes. Heartbeats now compute counts
in O(1).
Tests updated: the initial fixture pre-populates .ecj with two ids to
verify the file-size derivation end-to-end, and the delete test keeps
its idempotent-re-delete / missing-needle invariants (unchanged
externally, now enforced by the early return rather than a cache guard).
* ec: sync Rust volume server with Go file/delete count semantics
Mirror the Go-side EC file/delete count work in the Rust volume server
so mixed Go/Rust clusters report consistent bucket object counts in
the admin dashboard.
- Add file_count (8) and delete_count (9) to the Rust copy of
VolumeEcShardInformationMessage (seaweed-volume/proto/master.proto).
- EcVolume gains ecj_file_size, seeded from the journal's metadata on
open and bumped inside journal_delete on every successful append.
- file_and_delete_count() returns counts derived in O(1) from
ecx_file_size / NEEDLE_MAP_ENTRY_SIZE and
ecj_file_size / NEEDLE_ID_SIZE, matching Go's FileAndDeleteCount.
- to_volume_ec_shard_information_messages populates the new proto
fields instead of defaulting them to zero.
- mark_needle_deleted_in_ecx now returns a DeleteOutcome enum
(NotFound / AlreadyDeleted / Tombstoned) so journal_delete can skip
both the .ecj append and the size bump when the needle is missing
or already tombstoned, keeping the derived delete_count idempotent
on repeat or no-op deletes.
- Rust's EcVolume::new no longer replays .ecj into .ecx on load. Go's
RebuildEcxFile is only called from specific decode/rebuild gRPC
handlers, not on volume open, and replaying on load was hiding the
deletion journal from the new file-size-derived delete counter.
rebuild_ecx_from_journal is kept as dead_code for future decode
paths that may want the same replay semantics.
Also clean up the Go FileAndDeleteCount to drop unnecessary runtime
guards against zero constants — NeedleMapEntrySize and NeedleIdSize
are compile-time non-zero.
test_ec_volume_journal updated to pre-populate the .ecx with the
needles it deletes, and extended to verify that repeat and
missing-id deletes do not drift the derived counts.
* ec: document enterprise-reserved proto field range on ec shard info
Both OSS master.proto copies now note that fields 10-19 are reserved
for future upstream additions while 20+ are owned by the enterprise
fork. Enterprise already pins data_shards/parity_shards at 20/21, so
keeping OSS additions inside 8-19 avoids wire-level collisions for
mixed deployments.
* ec(rust): resolve .ecx/.ecj helpers from ecx_actual_dir
ecx_file_name() and ecj_file_name() resolved from self.dir_idx, but
new() opens the actual files from ecx_actual_dir (which may fall back
to the data dir when the idx dir does not contain the index). After a
fallback, read_deleted_needles() and rebuild_ecx_from_journal() would
read/rebuild the wrong (nonexistent) path while heartbeats reported
counts from the file actually in use — silently dropping deletes.
Point idx_base_name() at ecx_actual_dir, which is initialized to
dir_idx and only diverges after a successful fallback, so every call
site agrees with the file new() has open. The pre-fallback call in
new() (line 142) still returns the dir_idx path because
ecx_actual_dir == dir_idx at that point.
Update the destroy() sweep to build the dir_idx cleanup paths
explicitly instead of leaning on the helpers, so post-fallback stale
files in the idx dir are still removed.
* ec: reset ecj size after rebuild; rollback ecx tombstone on ecj failure
Two EC delete-count correctness fixes applied symmetrically to Go and
Rust volume servers.
1. rebuild_ecx_from_journal (Rust) now sets ecj_file_size = 0 after
recreating the empty journal, matching the on-disk truth.
Previously the cached size still reflected the pre-rebuild journal
and file_and_delete_count() would keep reporting stale delete
counts. The Go side has no equivalent bug because RebuildEcxFile
runs in an offline helper that does not touch an EcVolume struct.
2. DeleteNeedleFromEcx / journal_delete used to tombstone the .ecx
entry before writing the .ecj record. If the .ecj append then
failed, the needle was permanently marked deleted but the
heartbeat-reported delete_count never advanced (it is derived from
.ecj file size), and a retry would see AlreadyDeleted and early-
return, leaving the drift permanent.
Both languages now capture the entry's file offset and original
size bytes during the mark step, attempt the .ecj append, and on
failure roll the .ecx tombstone back by writing the original size
bytes at the known offset. A rollback that itself errors is
logged (glog / tracing) but cannot re-sync the files — this is
the same failure mode a double disk error would produce, and is
unavoidable without a full on-disk transaction log.
Go: wrap MarkNeedleDeleted in a closure that captures the file
offset into an outer variable, then pass the offset + oldSize to the
new rollbackEcxTombstone helper on .ecj seek/write errors.
Rust: DeleteOutcome::Tombstoned now carries the size_offset and a
[u8; SIZE_SIZE] copy of the pre-tombstone size field. journal_delete
destructures on Tombstoned and calls restore_ecx_size on .ecj append
failure.
* test(ec): widen admin /health wait to 180s for cold CI
TestEcEndToEnd starts master, 14 volume servers, filer, 2 workers and
admin in sequence, then waited only 60s for admin's HTTP server to come
up. On cold GitHub runners the tail of the earlier subprocess startups
eats most of that budget and the wait occasionally times out (last hit
on run 24374773031). The local fast path is still ~20s total, so the
bump only extends the timeout ceiling, not the happy path.
* test(ec): fork volume servers in parallel in TestEcEndToEnd
startWeed is non-blocking (just cmd.Start()), so the per-process fork +
mkdir + log-file-open overhead for 14 volume servers was serialized for
no reason. On cold CI disks that overhead stacks up and eats into the
subsequent admin /health wait, which is how run 24374773031 flaked.
Wrap the volume-server loop in a sync.WaitGroup and guard runningCmds
with a mutex so concurrent appends are safe. startWeed still calls
t.Fatalf on failure, which is fine from a goroutine for a fatal test
abort; the fail-fast isn't something we rely on for precise ordering.
* ec: fsync ecx before ecj, truncate on failure, harden rebuild
Four correctness fixes covering both volume servers.
1. Durability ordering (Go + Rust). After marking the .ecx tombstone
we now fsync .ecx before touching .ecj, so a crash between the two
files cannot leave the journal with an entry for a needle whose
tombstone is still sitting in page cache. Once the fsync returns,
the tombstone is the source of truth: reads see "deleted",
delete_count may under-count by one (benign, idempotent retries)
but never over-reports. If the fsync itself fails we restore the
original size bytes and surface the error. The .ecj append is then
followed by its own Sync so the reported delete_count matches the
on-disk journal once the write returns.
2. .ecj truncation on append failure. write_all may have extended the
journal on disk before sync_all / Sync errors out, leaving the
cached ecj_file_size out of sync with the physical length and
drifting delete_count permanently after restart. Both languages
now capture the pre-append size, truncate the file back via
set_len / Truncate on any write or sync failure, and only then
restore the .ecx tombstone. Truncation errors are logged — same-fd
length resets cannot realistically fail — but cannot themselves
re-sync the files.
3. Atomic rebuild_ecx_from_journal (Rust, dead code today but wired
up on any future decode path). Previously a failed
mark_needle_deleted_in_ecx call was swallowed with `let _ = ...`
and the journal was still removed, silently losing tombstones.
We now bubble up any non-NotFound error, fsync .ecx after the
whole replay succeeds, and only then drop and recreate .ecj.
NotFound is still ignored (expected race between delete and encode).
4. Missing-.ecx hardening (Rust). mark_needle_deleted_in_ecx used to
return Ok(NotFound) when self.ecx_file was None, hiding a closed or
corrupt volume behind what looks like an idempotent no-op. It now
returns an io::Error carrying the volume id so callers (e.g.
journal_delete) fail loudly instead.
Existing Go and Rust EC test suites stay green.
* ec: make .ecx immutable at runtime; track deletes in memory + .ecj
Refactors both volume servers so the sealed sorted .ecx index is never
mutated during normal operation. Runtime deletes are committed to the
.ecj deletion journal and tracked in an in-memory deleted-needle set;
read-path lookups consult that set to mask out deleted ids on top of
the immutable .ecx record. Mirrors the intended design on both Go and
Rust sides.
EcVolume gains a `deletedNeedles` / `deleted_needles` set seeded from
.ecj in NewEcVolume / EcVolume::new. DeleteNeedleFromEcx /
journal_delete:
1. Looks the needle up read-only in .ecx.
2. Missing needle -> no-op.
3. Pre-existing .ecx tombstone (from a prior decode/rebuild) ->
mirror into the in-memory set, no .ecj append.
4. Otherwise append the id to .ecj, fsync, and only then publish
the id into the set. A partial write is truncated back to the
pre-append length so the on-disk journal and the in-memory set
cannot drift.
FindNeedleFromEcx / find_needle_from_ecx now return
TombstoneFileSize when the id is in the in-memory set, even though
the bytes on disk still show the original size.
FileAndDeleteCount:
fileCount = .ecx size / NeedleMapEntrySize (unchanged)
deleteCount = len(deletedNeedles) (was: .ecj size / NeedleIdSize)
The RebuildEcxFile / rebuild_ecx_from_journal decode-time helpers
still fold .ecj into .ecx — that is the one place tombstones land in
the physical index, and it runs offline on closed files. Rust's
rebuild helper now also clears the in-memory set when it succeeds.
Dead code removed on the Rust side: `DeleteOutcome`,
`mark_needle_deleted_in_ecx`, `restore_ecx_size`. Go drops the
runtime `rollbackEcxTombstone` path. Neither helper was needed once
.ecx stopped being a runtime mutation target.
TestEcVolumeSyncEnsuresDeletionsVisible (issue #7751) is rewritten
as TestEcVolumeDeleteDurableToJournal, which exercises the full
durability chain: delete -> .ecj fsync -> FindNeedleFromEcx masks
via the in-memory set -> raw .ecx bytes are *unchanged* -> Close +
RebuildEcxFile folds the journal into .ecx -> raw bytes now show
the tombstone, as CopyFile in the decode path expects.
Mkdir was masking in.Mode with wfs.option.Umask on top of the kernel's
VFS umask pass, so a caller with umask=0 who requested mkdir(0777) got
0755 (0777 & ~022). Create and Symlink don't apply this second pass —
Mkdir was the odd one out. The resulting dirs had fewer write bits than
the caller asked for, which broke cross-user rename permission checks
(kernel may_delete rejects with EACCES when the parent lacks o+w even
though the caller explicitly requested it) and blocked pjdfstest
tests/rename/21.t and its cascading checks.
Drop the extra umask so Mkdir trusts in.Mode exactly like Create. The
CLI -umask flag still covers the internal cache dirs that the mount
creates for itself via os.MkdirAll; only the user-facing Mkdir path
changes.
Unblocks tests/rename/21.t — full pjdfstest suite is now 236 files /
8819 tests, all PASS, and known_failures.txt is empty.
* fix(mount): propagate hard-link nlink changes to sibling cache entries
weed mount serves stat from its local metacache, and the kernel also
caches inode attrs from FUSE replies. When a hard link was unlinked or
a new link added, the filer updated the shared HardLink blob correctly,
but the sibling link entries in the mount's metacache still carried the
stale HardLinkCounter and the kernel attr cache on the shared inode was
not invalidated. Subsequent lstat on any sibling link returned the old
nlink — pjdfstest link/00.t caught this after `unlink n0` and on
`link n1 n2` stating n0.
Walk every path bound to the hard-linked inode via a new
InodeToPath.GetAllPaths, rewrite each cached sibling's HardLinkCounter
and ctime to the authoritative new value, and call
fuseServer.InodeNotify to invalidate the kernel attr cache for the
shared inode. Applied from both Link (bump) and Unlink (decrement).
Unblocks tests/link/00.t and tests/unlink/00.t in pjdfstest; full suite
(235 files, 8803 tests) passes end-to-end with no regressions.
* fix(mount): harden hard-link sibling sync against nil Attributes and id mismatch
Review follow-ups:
- Unlink: guard entry.Attributes for nil before reading Inode, with a
fallback to inodeToPath.GetInode resolved before RemovePath. Fold the
duplicated RemovePath into a single call.
- syncHardLinkSiblings: skip siblings whose HardLinkId does not match
the authoritative entry. The shared-inode invariant normally
guarantees a match, but a transient mismatch (e.g. a rename replaced
one of the paths) would otherwise stamp an unrelated entry with the
wrong counter.
Full pjdfstest suite still passes (235 files, 8803 tests).
* test(vacuum): fix flaky TestVacuumIntegration across multiple volumes
The test assumed all uploaded files landed in a single volume and
tracked only the last file's volume id. With -volumeSizeLimitMB 10
and 16x500KB files, the master can spread uploads across volumes,
so the tracked id could point to a volume with no deletes and thus
0% garbage — causing verify_garbage_before_vacuum to fail even
though vacuum ran correctly on the other volume.
Track the set of volumes where deletes actually occurred and
verify garbage/cleanup against all of them. Also add a short
retry loop on the pre-vacuum check to absorb heartbeat jitter.
* test(vacuum): require all dirty volumes ready; retry cleanup check
Address review feedback: the pre-vacuum check now waits until every
volume in dirtyVolumes reports garbage > threshold (not just the
first), and the post-vacuum cleanup check retries per-volume with a
deadline instead of relying on a fixed sleep, since vacuum + heartbeat
reporting is asynchronous.
* test(vacuum): deterministic dirty volumes order, aggregate cleanup failures
- Sort dirtyVolumes after building from the set so logs and iteration
are stable across runs.
- In verify_cleanup_after_vacuum, track per-volume failure reasons in a
map and report all still-failing volumes on timeout instead of only
the last one that happened to be written to lastErr.
* docker: upgrade libcrypto3/libssl3 to clear Trivy HIGH
Trivy gate on ghcr.io/seaweedfs/seaweedfs:latest-amd64 flagged
CVE-2026-28390 in libcrypto3 3.5.5-r0 (fixed in 3.5.6-r0) on the
alpine 3.23.3 base. Add libcrypto3/libssl3 to the existing apk upgrade
so rebuilt images pick up the patched openssl without waiting for a
new alpine base tag.
* docker: apk add libcrypto3/libssl3 so they install at patched version
Per review, apk upgrade <pkg> is a no-op when the package isn't already
installed. libcrypto3/libssl3 come in transitively via curl, so list
them in apk add to guarantee installation at the latest (patched)
version from the alpine repo.
* admin: include EC volumes in bucket size reporting
The Object Store buckets page computed per-collection size by iterating
only regular volumes, so once a bucket's data was EC-encoded it silently
disappeared from the reported size — breaking usage-based billing.
Walk EcShardInfos alongside VolumeInfos in collectCollectionStats: add
raw shard bytes to PhysicalSize, and the parity-stripped value
(shardBytes * DataShardsCount / TotalShardsCount) to LogicalSize,
matching the normalization used by `weed shell` cluster.status.
* admin: derive EC logical size from shard bitmap, not constants
Use ShardsInfoFromVolumeEcShardInformationMessage + MinusParityShards
to sum actual data-shard bytes instead of scaling raw bytes by the
DataShardsCount/TotalShardsCount ratio. Keeps the data/parity split
encapsulated in the erasure_coding package and is exact when shard
sizes differ (e.g. last shard).
* admin: regression test for EC shard size aggregation
Cover the uneven-tail-shard case (data shard 9 < 1000 bytes) and the
empty-collection-name path to pin PhysicalSize/LogicalSize behavior
for collectCollectionStats against future changes.
* s3api: prune bucket-scoped IAM actions on DeleteBucket
DeleteBucket removed the bucket directory and collection but left
behind any identity actions configured via s3.configure that were
scoped to that bucket (e.g. Read:bucket, Write:bucket/prefix),
leaving stale auth metadata that users expected to be cleaned up
along with the bucket.
After a successful delete, strip actions whose resource is exactly
the bucket or a prefix under it, save via the credential manager,
and let the existing filer metadata subscription fan the reload out
to every S3 server. Wildcarded resources and global actions are
preserved since they may cover other buckets; static identities
are left untouched.
Fixes#5310
* s3api: address review feedback on bucket IAM prune
- Apply per-identity updates via credentialManager.UpdateUser instead
of a full LoadConfiguration/SaveConfiguration round-trip, so the
prune no longer clobbers concurrent IAM edits made by s3.configure
or the IAM API during a DeleteBucket.
- Use a 30s bounded background context for the post-delete cleanup so
it survives client disconnect — the bucket is already gone by then
and this is best-effort bookkeeping.
- Skip static identities via IsStaticIdentity, since the credential
store never persists them and UpdateUser would return NotFound.
* Update documentation for helm chart, with instructions on how to deploy the RocksDB image tag variant.
Signed-off-by: Mark McCormick <mark.mccormick@chainguard.dev>
Nit: Update example to make it clearer that the seaweedfs version needs to be replaced.
Signed-off-by: Mark McCormick <mark.mccormick@chainguard.dev>
* docs(helm): clarify RocksDB variant instructions
- Note that filer persistence (enablePVC) is required so RocksDB
metadata survives restarts.
- Explain why master/volume also use the rocksdb-tagged image.
- Tighten wording around WEED_LEVELDB2_ENABLED override.
---------
Signed-off-by: Mark McCormick <mark.mccormick@chainguard.dev>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(scheduler): give worker tasks a real per-attempt execution deadline
The plugin scheduler derived the per-attempt execution deadline as
DetectionTimeoutSeconds * 2, which capped every worker task at twice
the cluster-scan budget regardless of actual work. For volume_balance
batches this was 240s — far too short for 20 large volume copies, so
every attempt died at "context deadline exceeded" and all in-flight
sub-RPCs surfaced as "context canceled". Retries restarted from move 1
and hit the same wall.
Add an explicit ExecutionTimeoutSeconds field to the plugin proto and
make each handler declare its own baseline (1800s for vacuum, balance,
EC; 3600s for iceberg). Size-aware handlers also emit an
estimated_runtime_seconds parameter on each proposal so the scheduler
extends the per-attempt deadline based on actual workload:
- volume_balance batch: max(largest single move, total / concurrency)
at 5 min/GB, so a skewed batch with one big volume isn't averaged
away.
- volume_balance single, vacuum (already), erasure_coding (10 min/GB),
ec_balance (5 min/GB): per-volume budgets.
admin_script and iceberg keep the configurable handler default since
their workloads are opaque to the detector.
* fix(scheduler): apply descriptor defaults to existing persisted configs
The previous commit added execution_timeout_seconds to the proto and
each handler's descriptor defaults, but two paths still left existing
deployments broken:
1. deriveSchedulerAdminRuntime returned stored AdminRuntime configs
as-is. Persisted configs from older versions have no
execution_timeout_seconds, so the scheduler fell back to the 90s
default — worse than the prior 240s behavior. Overlay descriptor
defaults for any zero numeric fields when loading.
2. The admin form did not round-trip execution_timeout_seconds, so a
normal save would clear it back to zero. Add the input field, the
fillAdminSettings/collectAdminSettings hooks, and as defense in
depth reapply descriptor defaults in UpdatePluginJobTypeConfigAPI
before persisting so a stale form can never silently clobber a
baseline.
* fix(volume_balance): account for partial scheduling rounds in batch estimate
With N moves and C slots, the busiest slot processes ceil(N/C) moves,
not N/C. Dividing total seconds by C underestimates wall-clock time
whenever N is not a multiple of C — e.g. 6 moves at concurrency 5
needs 2 rounds, not 1.2. Use avg * ceil(N/C) so partial rounds are
counted as full ones.
* fix(volume_balance): scale minBudget per wave instead of per move
Orchestration overhead (setup/teardown for the parallel move runner)
happens once per wave, not once per move. Use numRounds*60 as the
floor instead of len(moves)*60 so the minimum doesn't inflate
linearly with batch size when individual moves are tiny.
* fix(admin): allow control chars in file paths when browsing filer
The admin UI rejected any path containing \x00, \r, or \n as "path contains
invalid characters". These bytes are legal in S3 object keys, so objects
created through the S3 API (or replicated via filer.sync) could exist on the
filer but be unreachable from the admin UI — browse, download, and upload
all failed with "Invalid file path".
Drop the control-character rejection and instead URL-escape the path when
constructing filer request URLs, so that such bytes cannot inject into the
HTTP request target. Path traversal protection via path.Clean is unchanged.
* test(admin): strengthen file path tests with byte-preserving checks
Assert full expected output for validateAndCleanFilePath so silent stripping
of control characters would fail the test, and cover \r and \x00 escaping in
filerFileURL in addition to \n and space.
* fix(filer): eliminate redundant disk reads causing memory/CPU regression (#9035)
Since 4.18, LocalMetaLogBuffer's ReadFromDiskFn was set to
readPersistedLogBufferPosition, causing LoopProcessLogData to call
ReadPersistedLogBuffer on every 250ms health-check tick when a
subscriber encounters ResumeFromDiskError. Each call creates an
OrderedLogVisitor (ListDirectoryEntries on the filer store), spawns a
readahead goroutine with a 1024-element channel, finds no data, and
returns — 4 times per second even on an idle filer.
This is redundant because SubscribeLocalMetadata already manages disk
reads explicitly with its own shouldReadFromDisk / lastCheckedFlushTsNs
tracking in the outer loop.
Set ReadFromDiskFn back to nil for LocalMetaLogBuffer. When
LoopProcessLogData encounters ResumeFromDiskError with nil
ReadFromDiskFn, the HasData() guard returns ResumeFromDiskError to the
caller (SubscribeLocalMetadata), which blocks efficiently on
listenersCond.Wait() instead of polling.
* fix(filer): add gap detection for slow consumers after disk-read stall
When a slow consumer falls behind and LoopProcessLogData returns
ResumeFromDiskError with no flush or read-position progress, there may
be a gap between persisted data and in-memory data (e.g. writes stopped
while consumer was still catching up). Without this, the consumer would
block on listenersCond.Wait() forever.
Skip forward to the earliest in-memory time to resume progress, matching
the gap-handling pattern already used in the shouldReadFromDisk path.
* fix(filer): clear stale ResumeFromDiskError after gap-skip to avoid stall
The gap-detection block added in the previous commit skips lastReadTime
forward to GetEarliestTime() and continues the outer loop. On the next
iteration, shouldReadFromDisk becomes true (currentReadTsNs >
lastDiskReadTsNs), the disk read returns processedTsNs == 0, and the
existing gap handler at the top of the loop runs its own gap check.
That check uses readInMemoryLogErr == ResumeFromDiskError as the entry
condition — but readInMemoryLogErr is still the stale error from two
iterations ago. GetEarliestTime() now equals lastReadTime.Time (we
already advanced to it), so earliestTime.After(lastReadTime.Time) is
false and the handler falls into listenersCond.Wait() — stuck.
Clear readInMemoryLogErr at the gap-skip point, matching the existing
pattern at the earlier gap handler that already clears it for the same
reason.
* fix(log_buffer): GetEarliestTime must include sealed prev buffers
GetEarliestTime previously returned only logBuffer.startTime (the active
buffer's first timestamp). That is narrower than ReadFromBuffer's
tsMemory, which is the min across active + prev buffers. Callers using
GetEarliestTime for gap detection after ResumeFromDiskError (the
SubscribeLocalMetadata outer loop's disk-read path, the new gap-skip in
the in-memory ResumeFromDiskError handler, and MQ HasData) saw a time
that was *newer* than the real earliest in-memory data.
Impact in SubscribeLocalMetadata's slow-consumer path:
- tsMemory = earliest prev buffer time (T_prev)
- GetEarliestTime() = active startTime (T_active, later than T_prev)
- Consumer position = T1, with T_prev < T1 < T_active
- ReadFromBuffer returns ResumeFromDiskError (T1 < tsMemory)
- Gap detect: GetEarliestTime().After(T1) = T_active.After(T1) = true
- Skip forward to T_active -- silently drops the prev-buffer data
- And when T_active happens to equal the stuck position, gap detect
evaluates false, and the subscriber stalls on listenersCond.Wait()
This reproduces the TestMetadataSubscribeSlowConsumerKeepsProgressing
failure in CI where the consumer stalled at 10220/20000 after writing
stopped -- the buffer still had data in prev[0..3], but gap detection
was comparing against the active buffer's startTime.
Fix: scan all sealed prev buffers under RLock, return the true minimum
startTime. Matches the min-of-buffers logic in ReadFromBuffer.
* test(log_buffer): make DiskReadRetry test deterministic
The previous test added the message via AddToBuffer + ForceFlush and
relied on a race: the second disk read had to happen before the data
was delivered through the in-memory path. Under the race detector or
on a slow CI runner, the reader is woken by AddToBuffer's notification,
finds the data in the active buffer or its prev slot, and returns after
exactly one disk read — failing the >= 2 disk reads assertion even
though the loop behaved correctly.
Reproduced on master with race detector (2/5 failures).
Rewrite the test to deliver the data exclusively through the disk-read
path: no AddToBuffer, no ForceFlush. The test waits until the reader
has issued at least one no-op disk read, then atomically flips a
"dataReady" flag. The reader's next iteration through readFromDiskFn
returns the entry. This deterministically exercises the retry-loop
behavior the test was originally written to protect, and removes the
in-memory delivery race entirely.
* fix(shell): s3.user.provision handles existing users by attaching policy
Instead of erroring when the user already exists, the command now
creates the policy and attaches it to the existing user via UpdateUser.
Credentials are only generated and displayed for newly created users.
* fix(shell): skip duplicate policy attachment in s3.user.provision
Check if the policy is already attached before appending and calling
UpdateUser, making repeated runs idempotent.
* fix(shell): generate service account ID in s3.serviceaccount.create
The command built a ServiceAccount proto without setting Id, which was
rejected by credential.ValidateServiceAccountId on any real store. Now
generates sa:<parent>:<uuid> matching the format used by the admin UI.
* test(s3): integration tests for s3.* shell commands
Adds TestShell* integration tests covering ~40 previously untested
shell commands: user, accesskey, group, serviceaccount, anonymous,
bucket, policy.attach/detach, config.show, and iam.export/import.
Switches the test cluster's credential store from memory to filer_etc
because the memory store silently drops groups and service accounts
in LoadConfiguration/SaveConfiguration.
* fix(shell): rollback policy on key generation failure in s3.user.provision
If iam.GenerateRandomString or iam.GenerateSecretAccessKey fails after
the policy was persisted, the policy would be left orphaned. Extracts
the rollback logic into a local closure and invokes it on all failure
paths after policy creation for consistency.
* address PR review feedback for s3 shell tests and serviceaccount
- s3.serviceaccount.create: use 16 bytes of randomness (hex-encoded) for
the service account UUID instead of 4 bytes to eliminate collision risk
- s3.serviceaccount.create: print the actual ID and drop the outdated
"server-assigned" note (the ID is now client-generated)
- tests: guard createdAK in accesskey rotate/delete subtests so sibling
failures don't run invalid CLI calls
- tests: requireContains/requireNotContains use t.Fatalf to fail fast
- tests: Provision subtest asserts the "Attached policy" message on the
second provision call for an existing user
- tests: update extractServiceAccountID comment example to match the
sa:<parent>:<uuid> format
- tests: drop redundant saID empty-check (extractServiceAccountID fatals)
* test(s3): use t.Fatalf for precondition check in serviceaccount test
* fix: wait for in-flight uploads to complete before filer shutdown
Prevents data corruption when SIGTERM is received during active uploads.
The filer now waits for all in-flight operations to complete before
calling the underlying shutdown logic.
This affects all deployment types (Kubernetes, Docker, systemd) and
fixes corruption issues during rolling updates, certificate rotation,
and manual restarts.
Changes:
- Add FilerServer.Shutdown() method with upload wait logic
- Update grace.OnInterrupt hook to use new shutdown method
Fixes data corruption reported by production users during pod restarts.
* fix: implement graceful shutdown for gRPC and HTTP servers, ensuring in-flight uploads complete
* fix: address review comments on graceful shutdown
- Add 10s timeout to gRPC GracefulStop to prevent indefinite blocking
from long-lived streams (falls back to Stop on timeout)
- Reduce HTTP/HTTPS shutdown timeout from 25s to 15s to fit within
Kubernetes default 30s termination grace period
- Move fs.Shutdown() (database close) after Serve() returns instead
of a separate hook to eliminate race where main goroutine exits
before the shutdown hook runs
* fix: shut down all HTTP servers before filer database close
Address remaining review comments:
- Shut down auxiliary HTTP servers (Unix socket, local listener) during
graceful shutdown so they can't serve write traffic after the main
server stops
- Register fs.Shutdown() as a grace.OnInterrupt hook to guarantee it
completes before os.Exit(0), fixing the race between the grace
goroutine and the main goroutine
- Use sync.Once to ensure fs.Shutdown() runs exactly once regardless
of whether shutdown is signal-driven or context-driven (MiniCluster)
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat(mount): pre-allocate file IDs in pool for writeback cache mode
When writeback caching is enabled, chunk uploads no longer block on a
per-chunk AssignVolume RPC. Instead, a FileIdPool pre-allocates file IDs
in batches using a single AssignVolume(Count=N, ExpectedDataSize=ChunkSize)
call and hands them out instantly to upload workers.
Pool size is 2x ConcurrentWriters, refilled in background when it drops
below ConcurrentWriters. Entries expire after 25s to respect JWT TTL.
Sequential needle keys are generated from the base file ID returned by
the master, so one Assign RPC produces N usable IDs.
This cuts per-chunk upload latency from 2 RTTs (assign + upload) to
1 RTT (upload only), with the assign cost amortized across the batch.
* test: add benchmarks for file ID pool vs direct assign
Benchmarks measure:
- Pool Get vs Direct AssignVolume at various simulated latencies
- Batch assign scaling (Count=1 through Count=32)
- Concurrent pool access with 1-64 workers
Results on Apple M4:
- Pool Get: constant ~3ns regardless of assign latency
- Batch=16: 15.7x more IDs/sec than individual assigns
- 64 concurrent workers: 19M IDs/sec throughput
* fix(mount): address review feedback on file ID pool
1. Fix race condition in Get(): use sync.Cond so callers wait for an
in-flight refill instead of returning an error when the pool is empty.
2. Match default pool size to async flush worker count (128, not 16)
when ConcurrentWriters is unset.
3. Add logging to UploadWithAssignFunc for consistency with UploadWithRetry.
4. Document that pooled assigns omit the Path field, bypassing path-based
storage rules (filer.conf). This is an intentional tradeoff for
writeback cache performance.
5. Fix flaky expiry test: widen time margin from 50ms to 1s.
6. Add TestFileIdPoolGetWaitsForRefill to verify concurrent waiters.
* fix(mount): use individual Count=1 assigns to get per-fid JWTs
The master generates one JWT per AssignResponse, bound to the base file
ID (master_grpc_server_assign.go:158). The volume server validates that
the JWT's Fid matches the upload exactly (volume_server_handlers.go:367).
Using Count=N and deriving sequential IDs would fail this check.
Switch to individual Count=1 RPCs over a single gRPC connection. This
still amortizes connection overhead while getting a correct per-fid JWT
for each entry. Partial batches are accepted if some requests fail.
Remove unused needle import now that sequential ID generation is gone.
* fix(mount): separate pprof from FUSE protocol debug logging
The -debug flag was enabling both the pprof HTTP server and the noisy
go-fuse protocol logging (rx/tx lines for every FUSE operation). This
makes profiling impractical as the log output dominates.
Split into two flags:
- -debug: enables pprof HTTP server only (for profiling)
- -debug.fuse: enables raw FUSE protocol request/response logging
* perf(mount): replace LevelDB read+write with in-memory overlay for dir mtime
Profile showed TouchDirMtimeCtime at 0.22s — every create/rename/unlink
in a directory did a LevelDB FindEntry (read) + UpdateEntry (write) just
to bump the parent dir's mtime/ctime.
Replace with an in-memory map (same pattern as existing atime overlay):
- touchDirMtimeCtimeLocal now stores inode→timestamp in dirMtimeMap
- applyInMemoryDirMtime overlays onto GetAttr/Lookup output
- No LevelDB I/O on the mutation hot path
The overlay only advances timestamps forward (max of stored vs overlay),
so stale entries are harmless. Map is bounded at 8192 entries.
* perf(mount): skip self-originated metadata subscription events in writeback mode
With writeback caching, this mount is the single writer. All local
mutations are already applied to the local meta cache (via
applyLocalMetadataEvent or direct InsertEntry). The filer subscription
then delivers the same event back, causing redundant work:
proto.Clone, enqueue to apply loop, dedup ring check, and sometimes
redundant LevelDB writes when the dedup ring misses (deferred creates).
Check EventNotification.Signatures against selfSignature and skip
events that originated from this mount. This eliminates the redundant
processing for every self-originated mutation.
* perf(mount): increase kernel FUSE cache TTL in writeback cache mode
With writeback caching, this mount is the single writer — the local
meta cache is authoritative. Increase EntryValid and AttrValid from 1s
to 10s so the kernel doesn't re-issue Lookup/GetAttr for every path
component and stat call.
This reduces FUSE /dev/fuse round-trips which dominate the profile at
38% of CPU (syscall.rawsyscalln). Each saved round-trip eliminates a
kernel→userspace→kernel transition.
Normal (non-writeback) mode retains the 1s TTL for multi-mount
consistency.
* feat(master): drain pending size before marking volume readonly
When vacuum, volume move, or EC encoding marks a volume readonly,
in-flight assigned bytes may still be pending. This adds a drain step:
immediately remove from writable list (stop new assigns), then wait
for pending to decay below 4MB or 30s timeout.
- Add volumeSizeTracking struct consolidating effectiveSize,
reportedSize, and compactRevision into a single map
- Add GetPendingSize, waitForPendingDrain, DrainAndRemoveFromWritable,
DrainAndSetVolumeReadOnly to VolumeLayout
- UpdateVolumeSize detects compaction via compactRevision change and
resets effectiveSize instead of decaying
- Wire drain into vacuum (topology_vacuum.go) and volume mark readonly
(master_grpc_server_volume.go)
* fix: use 2MB pending size drain threshold
* fix: check crowded state on initial UpdateVolumeSize registration
* fix: respect context cancellation in drain, relax test timing
- DrainAndSetVolumeReadOnly now accepts context.Context and returns
early on cancellation (for gRPC handler timeout/cancel)
- waitForPendingDrain uses select on ctx.Done instead of time.Sleep
- Increase concurrent heartbeat test timeout from 10s to 15s for CI
* fix: use time-based dedup so decay runs even when reported size is unchanged
The value-based dedup (same reportedSize + compactRevision = skip) prevented
decay from running when pending bytes existed but no writes had landed on
disk yet. The reported size stayed the same across heartbeats, so the excess
never decayed.
Fix: dedup replicas within the same heartbeat cycle using a 2-second time
window instead of comparing values. This allows decay to run once per
heartbeat cycle even when the reported size is unchanged.
Also confirmed finding 1 (draining re-add race) is a false positive:
- Vacuum: ensureCorrectWritables only runs for ReadOnly-changed volumes
- Move/EC: readonlyVolumes flag prevents re-adding during drain
* fix: make VolumeMarkReadonly non-blocking to fix EC integration test timeout
The DrainAndSetVolumeReadOnly call in VolumeMarkReadonly gRPC blocked up
to 30s waiting for pending bytes to decay. In integration tests (and
real clusters during EC encoding), this caused timeouts because multiple
volumes are marked readonly sequentially and heartbeats may not arrive
fast enough to decay pending within the drain window.
Fix: VolumeMarkReadonly now calls SetVolumeReadOnly immediately (stops
new assigns) and only logs a warning if pending bytes remain. The drain
wait is kept only for vacuum (DrainAndRemoveFromWritable) which runs
inside the master's own goroutine pool.
Remove DrainAndSetVolumeReadOnly as it's no longer used.
* fix: relax test timing, rename test, add post-condition assert
* test: add vacuum integration tests with CI workflow
Full-cluster integration test for vacuum, modeled on the EC integration
tests. Starts a real master + 2 volume servers, uploads data, deletes
entries to create garbage, runs volume.vacuum via shell command, and
verifies garbage cleanup and data integrity.
Test flow:
1. Start cluster (master + 2 volume servers)
2. Upload 10 files to create volume with data
3. Delete 5 files to create ~50% garbage
4. Verify garbage ratio > 10%
5. Run volume.vacuum command
6. Verify garbage cleaned up
7. Verify remaining 5 files are still accessible
CI workflow runs on push/PR to master with 15-minute timeout.
Log collection on failure via artifact upload.
* fix: use 500KB files and delete 75% to exceed vacuum garbage threshold
* fix: add shell lock before vacuum command, fix compilation error
* fix: strengthen vacuum integration test assertions
- waitForServer: use net.DialTimeout instead of grpc.NewClient for
real TCP readiness check
- verify_garbage_before_vacuum: t.Fatal instead of warning when no
garbage detected
- verify_cleanup_after_vacuum: t.Fatal if no server reported the
volume or cleanup wasn't verified
- verify_remaining_data: read actual file contents via HTTP and
compare byte-for-byte against original uploaded payloads
* fix: use http.Client with timeout and close body before retry
* feat: pass expected_data_size from clients for size-aware assignment
Add expected_data_size field to AssignRequest (master proto) and
AssignVolumeRequest (filer proto) so clients can hint how large the
data will be. The master uses this instead of the 1MB default when
tracking pending volume sizes for weighted assignment.
- Add expected_data_size to master.proto AssignRequest
- Add expected_data_size to filer.proto AssignVolumeRequest
- Wire through filer AssignVolume handler
- Wire through HTTP submit handler (uses actual upload size)
- Add ExpectedDataSize to VolumeAssignRequest in operation package
- Topology.PickForWrite accepts optional expectedDataSize parameter
* fix: guard integer conversions in expected_data_size path
- common.go: clamp OriginalDataSize to non-negative before uint64 cast
- topology.go: cap expectedDataSize at math.MaxInt64 before int64 cast
* fix: parse dataSize hint in HTTP /dir/assign and test non-zero expectedDataSize
- HTTP /dir/assign now parses optional "dataSize" query parameter
and passes it to PickForWrite instead of hardcoded 0
- Add test assertion for PickForWrite with non-zero expectedDataSize
2026-04-11 11:30:47 -07:00
Chris LuGitHubCopilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* feat(master): size-aware volume assignment with weighted selection
PickForWrite now selects volumes proportional to remaining capacity
instead of uniform random, so emptier volumes receive more writes.
- Add vid2size map to VolumeLayout tracking effective volume sizes
- Weighted pick via random sampling (k=3) for O(1) cost
- RecordAssign tracks estimated pending bytes between heartbeats
- Exponential decay on heartbeat: halve excess each cycle
- Proactive crowded detection using effective size
- Zero extra heap allocations on the unconstrained hot path
Benchmark (20 writable volumes, unconstrained):
Before: 36 ns/op, 32 B/op, 2 allocs/op
After: 85 ns/op, 32 B/op, 2 allocs/op
* fix: address review feedback on size-aware assignment
- RecordAssign: use write lock (Lock) instead of read lock (RLock)
since it mutates vid2size map and crowded set
- RegisterVolume: clear crowded flag when heartbeat decay drops
effective size below the threshold
- pickWeightedByRemaining: fix misleading Fisher-Yates comment,
simplify to plain random sampling (duplicates are harmless)
- ShouldGrowVolumesByDcAndRack: read vid2size under RLock
* fix: decay once per heartbeat cycle, not per replica
RegisterVolume is called once per replica of a volume. For replicated
volumes, the pending size decay was running multiple times per heartbeat
cycle, reducing the excess by 75% instead of 50% (for 2 replicas).
Fix: track vid2reportedSize and only run decay when the heartbeat-
reported size actually changes. A second replica reporting the same
size in the same cycle is a no-op.
Also fix CodeQL alert: cap count*EstimatedNeedleSizeBytes to avoid
uint64→int64 overflow in RecordAssign call.
* Potential fix for pull request finding 'CodeQL / Incorrect conversion between integer types'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix: fail fast in test setup on JSON errors
- setupWithLimit now takes testing.TB and calls t.Fatalf on unmarshal
errors or type assertion failures instead of printing and continuing
- benchSetup removed; benchmarks reuse setupWithLimit directly
* fix: run size decay on every heartbeat, not just new volumes
RegisterVolume is only called for newly discovered volumes, not on
every heartbeat. The pending size decay was never running in production.
- Extract decay logic into UpdateVolumeSize(), called from
SyncDataNodeRegistration for every reported volume on every heartbeat
- RegisterVolume only initializes vid2size for brand-new volumes
- Constrained PickForWrite: scan from random offset, collect up to
pickSampleSize matches in a stack array (no append allocation)
- Tests now exercise UpdateVolumeSize directly instead of RegisterVolume
to match the production heartbeat path
* fix: compute pending bytes in uint64 to satisfy CodeQL
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix(mount): reduce filer RPCs for mkdir/rmdir operations
1. Mark newly created directories as cached immediately. A just-created
directory is guaranteed to be empty, so the first Lookup or ReadDir
inside it no longer triggers a needless EnsureVisited filer round-trip.
2. Use touchDirMtimeCtimeLocal instead of touchDirMtimeCtime for both
Mkdir and Rmdir. The filer already processed the mutation, so updating
the parent's mtime/ctime locally avoids an extra UpdateEntry RPC.
Net effect: mkdir goes from 3 filer RPCs to 1.
* fix(mount): eliminate extra filer RPCs for parent dir mtime updates
Every mutation (create, unlink, symlink, link, rename) was calling
touchDirMtimeCtime after the filer already processed the mutation.
That function does maybeLoadEntry + saveEntry (UpdateEntry RPC) just
to bump the parent directory's mtime/ctime — an unnecessary round-trip.
Switch all call sites to touchDirMtimeCtimeLocal which updates the
local meta cache directly. Remove the now-unused touchDirMtimeCtime.
Affected operations: Create (Mknod path), Unlink, Symlink, Link, Rename.
Each saves one filer RPC per call.
* fix(mount): defer RemoveXAttr for open files, skip redundant existence check
1. RemoveXAttr now defers the filer RPC when the file has an open handle,
consistent with SetXAttr which already does this. The xattr change is
flushed with the file metadata on close.
2. Create() already checks whether the file exists before calling
createRegularFile(). Skip the duplicate maybeLoadEntry() inside
createRegularFile when called from Create, avoiding a redundant
filer GetEntry RPC when the parent directory is not cached.
* fix(mount): skip distributed lock when writeback caching is enabled
Writeback caching implies single-writer semantics — the user accepts
that only one mount writes to each file. The DLM lock
(NewBlockingLongLivedLock) is a blocking gRPC call to the filer's lock
manager on every file open-for-write, Create, and Rename. This is
unnecessary overhead when writeback caching is on.
Skip lockClient initialization when WritebackCache is true. All DLM
call sites already guard on `wfs.lockClient != nil`, so they are
automatically skipped.
* fix(mount): async filer create for Mknod with writeback caching
With writeback caching, Mknod now inserts the entry into the local
meta cache immediately and fires the filer CreateEntry RPC in a
background goroutine, similar to how Create defers its filer RPC.
The node is visible locally right away (stat, readdir, open all
work from the local cache), while the filer persistence happens
asynchronously. This removes the synchronous filer RPC from the
Mknod hot path.
* fix(mount): address review feedback on async create and DLM logging
1. Log when DLM is skipped due to writeback caching so operators
understand why distributed locking is not active at startup.
2. Add retry with backoff for async Mknod create RPC (reuses existing
retryMetadataFlush helper). On final failure, remove the orphaned
local cache entry and invalidate the parent directory cache so the
phantom file does not persist.
* fix(mount): restore filer RPC for parent dir mtime when not using writeback cache
The local-only touchDirMtimeCtimeLocal updates LevelDB but lookupEntry
only reads from LevelDB when the parent directory is cached. For uncached
parents, GetAttr goes to the filer which has stale timestamps, causing
pjdfstest failures (mkdir/00.t, rmdir/00.t, unlink/00.t, etc.).
Introduce touchDirMtimeCtimeBest which:
- WritebackCache mode: local meta cache only (no filer RPC)
- Normal mode: filer UpdateEntry RPC for POSIX correctness
The deferred file create path keeps touchDirMtimeCtimeLocal since no
filer entry exists yet.
* fix(mount): use touchDirMtimeCtimeBest for deferred file create path
The deferred create path (Create with deferFilerCreate=true) was using
touchDirMtimeCtimeLocal unconditionally, but this only updates the local
LevelDB cache. Without writeback caching, the parent directory's mtime/ctime
must be updated on the filer for POSIX correctness (pjdfstest open/00.t).
* test: add link/00.t and unlink/00.t to pjdfstest known failures
These tests fail nlink assertions (e.g. expected nlink=2, got nlink=3)
after hard link creation/removal. The failures are deterministic and
surfaced by caching changes that affect the order in which entries are
loaded into the local meta cache. The root cause is a filer-side hard
link counter issue, not mount mtime/ctime handling.
Fix an issue where seleting Sepecific Buckets with Admin permission
while creating/editing an object store user would grant Admin permission on all
buckets
* fix(s3): preserve exact policy document in embedded IAM PutUserPolicy/GetUserPolicy (#9008)
The embedded IAM implementation (used when IAM requests go through the
S3 gateway) discarded the original policy document on PutUserPolicy,
storing only the lossy ident.Actions representation. GetUserPolicy then
reconstructed the document from these coarse-grained actions, producing
wildcard-expanded actions (s3:GetObject → s3:Get*), duplicates, and
collapsed resources (array → single string).
PR #9009 fixed this in the standalone IAM server (weed/iamapi/) but the
embedded IAM (weed/s3api/) — which is the code path most users hit —
had the same bugs.
Changes:
- Add InlinePolicyStore optional interface to credential store, with
implementations for FilerEtcStore (uses existing PoliciesCollection),
MemoryStore, and PropagatingCredentialStore.
- Embedded IAM PutUserPolicy now persists the original policy document
via CredentialManager.PutUserInlinePolicy for lossless round-trips.
- Embedded IAM GetUserPolicy first tries the stored inline policy; only
falls back to lossy reconstruction from ident.Actions when no stored
document exists (e.g. policies created before this fix).
- Fix the fallback reconstruction: add action deduplication and preserve
resource paths verbatim (no more spurious /* appending).
- Update DeleteUserPolicy/ListUserPolicies to use stored inline policies.
* fix(s3): address PR review feedback for embedded IAM inline policies
- Validate PolicyName is non-empty in PutUserPolicy and DeleteUserPolicy
- Add recomputeActions() to aggregate ident.Actions from ALL stored
inline policies on put/delete, fixing the issue where a second
PutUserPolicy would overwrite the first policy's enforcement
- Log errors from GetUserInlinePolicy in the GetUserPolicy fallback
instead of silently ignoring them
- Add initialization guards to MemoryStore GetUserInlinePolicy and
ListUserInlinePolicies for consistency with other read methods
* fix(s3): make inline policy persistence fatal and propagate recompute errors
Address second round of review feedback:
- recomputeActions() now returns ([]string, error) so callers can
distinguish store failures from "no stored policies" and abort the
mutation on transient errors instead of silently falling back.
- PutUserInlinePolicy and DeleteUserInlinePolicy failures are now fatal:
the API call returns ServiceFailure instead of logging and continuing,
keeping ident.Actions and stored policy state in sync.
* chore: gofmt weed/s3api/iceberg/handlers_oauth.go
Pre-existing formatting issue from #9017; fixes S3 Tables Format Check CI.
Track subdirectory count per-inode in memory via InodeEntry.subdirCount.
Increment on mkdir, decrement on rmdir, adjust on cross-directory
rename. applyDirNlink uses this count instead of listing metacache
entries, so nlink is correct immediately after mkdir without needing
a prior readdir.
Remove tests/rename/24.t from known_failures.txt (all 13 subtests
now pass).
fix(mount): skip metadata flush for unlinked-while-open files
When a file is unlinked while still open (open-unlink-close pattern),
the synchronous doFlush path recreated the entry on the filer during
close. Check fh.isDeleted before flushing metadata, matching the
existing check in the async flush path.
Remove tests/unlink/14.t from known_failures.txt (all 7 subtests
now pass). Full suite: 235 files, 8803 tests, Result: PASS.
When a file is unlinked while still open (open-unlink-close pattern),
the synchronous doFlush path would recreate the entry on the filer
during close. Check fh.isDeleted before flushing metadata, matching
the async flush path which already had this check.
The upstream pjd/pjdfstest uses hardcoded ~768-byte filenames which
exceed the Linux FUSE kernel NAME_MAX=255 limit. The sanwan fork
(used by JuiceFS) uses pathconf(_PC_NAME_MAX) to dynamically
determine the filesystem's actual NAME_MAX and generates test names
accordingly.
This removes all 26 NAME_MAX-related entries from known_failures.txt,
reducing the skip list from 31 to 5 entries.
The directory nlink counting (2 + subdirectory count) requires listing
cached directory entries on every stat, which has a performance cost.
Gate it behind the -posix.dirNLink flag (default: off).
When disabled, directories report nlink=2 (POSIX baseline).
When enabled, directories report nlink=2 + number of subdirectories
from cached entries.
fix(mount): report correct nlink for directories (2 + subdirectory count)
POSIX requires directory nlink = 2 (for . and ..) + number of
subdirectories. Previously SeaweedFS reported nlink=1 for all dirs.
- Set nlink baseline to 2 for directories in setAttrByPbEntry,
setAttrByFilerEntry, and setRootAttr
- Add applyDirNlink() that counts subdirectories from the local
metacache and sets nlink = 2 + count
- Call it from GetAttr and Lookup for directory entries
When the metacache has no entries (before readdir), nlink=2 is used
as a safe POSIX-compliant default.
When unlinking a hard-linked file, DeleteOneEntry and DeleteEntry both
called DeleteHardLink before removing the directory entry from the
store. If DeleteHardLink returned an error (e.g. KV storage issue,
decode failure), the function returned early without deleting the
directory entry itself. This left a stale entry in the filer store,
causing subsequent rmdir to fail with ENOTEMPTY.
Change both functions to log the hard link cleanup error and continue
to delete the directory entry regardless. This ensures the parent
directory can always be removed after all its children are unlinked.
Remove tests/unlink/14.t from the pjdfstest known failures list since
this fix addresses the root cause.
fix(filer): fix hard link nlink/ctime when rename replaces a hard-linked target
The CreateEntry → UpdateEntry → handleUpdateToHardLinks path already
calls DeleteHardLink() when the existing target has a different
HardLinkId. Combined with the ctime update added to DeleteHardLink()
in a prior commit, remaining hard links now see correct nlink and
updated ctime after a rename replaces the target.
Remove tests/rename/23.t and tests/rename/24.t from known_failures.txt.
t.Fatalf("GET /v1/config?warehouse=%s: overrides.prefix = %q, want %q (catalog must echo the warehouse's table bucket so clients like DuckDB know which /v1/{prefix}/namespaces to use)",
t.Fatalf("GET /v1/namespaces?warehouse=%s status = %d, want 200 (server ignores the warehouse query parameter and falls back to the default %q bucket, causing DuckDB to see no schemas)",
bareWarehouse,status,"warehouse")
case!containsNamespace(bareBody,namespace):
t.Fatalf("GET /v1/namespaces?warehouse=%s did not return %q: %v (server must honor the warehouse query parameter here or vend overrides.prefix from /v1/config so the client can route to the right bucket)",
bareWarehouse,namespace,bareBody)
}
}
// TestIssue9103_DuckDBAttachCannotResolveNamespace is the end-to-end
// reproduction of the issue using DuckDB in Docker, mirroring the exact
// sequence of commands from the bug report. It runs under its own weed
// mini with IAM configured so the OAuth2 client_credentials flow that
// DuckDB's iceberg extension requires actually works (the shared env has
// no credentials registered). Gated on Docker availability.
LogicalSizeint64`json:"logical_size"`// Actual data size (used space)
PhysicalSizeint64`json:"physical_size"`// Total allocated volume space
ObjectCountint64`json:"object_count"`
LastModifiedtime.Time`json:"last_modified"`
Quotaint64`json:"quota"`// Quota in bytes, 0 means no quota
QuotaEnabledbool`json:"quota_enabled"`// Whether quota is enabled
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Chris Lu
Chris LuandGitHub
Lars LehtonenandGitHub
Arthur WoimbéeandGitHub
Moray BaruhandGitHub