mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-07-20 06:52:24 +00:00
master
1973 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
564803becd |
shell: show who holds the cluster lock (#10353)
* regenerate master_grpc.pb.go with protoc-gen-go-grpc v1.6.2 The other generated pb files are already on v1.6.2; this one was stale. * shell: keep unlock from racing the lease renewal A renewal RPC in flight while ReleaseLock runs re-creates the lock on the master after the release deletes it, and can blank the client name if the renewal reads it mid-release. The stale-token release is then ignored, so the lock stays held (sometimes anonymously) until it expires. Serialize the renew and release RPCs, and set the client name before flipping isLocked so the renewal never sends a partial acquisition. * shell: restart lease renewal after a failed renewal The renewal goroutine exits on error but never cleared its running flag, so later locks in the same process were never renewed and silently expired after ten seconds. * shell: show who holds the cluster lock A blocked lock command gave no hint that another client holds the lock (the refusals only surfaced at -v=2), and cluster.status reported the shell's own lock state as if it were the cluster's. Add a GetAdminLockStatus RPC to the master so lock prints the holder before blocking and cluster.status shows the actual cluster-wide holder. Both degrade silently against masters without the RPC. * shell: bound admin lock RPC attempts with timeouts The lease, renew, release, and holder-status calls all ran without a deadline, so an unresponsive master could hang the renewal goroutine, an unlock (which now waits on the renewal mutex), or the shell prompt. Give each attempt its own short context; the retry loops still resolve a fresh leader on the next try. * master: reject admin token release on non-leaders A follower holds no lock state, so it answered a release with success while the leader kept the lock until expiry. Refuse like LeaseAdminToken does so the client can try the leader instead. * shell: leave the lock release call unbounded A release cut short by a deadline leaves the lock held on the master until it expires, so a slow master would turn every unlock into a ten-second ghost lock. Restore the single fire-and-forget attempt; the timeouts stay on the lease and renew paths, where a stalled call forfeits the lease anyway. * shell: release only the token unlock started with A RequestLock racing a slow release (the admin presence lock does this on shutdown) could have its freshly acquired token sent in the release request or zeroed by the trailing stores. Capture the token once under the mutex and compare on clear so a concurrent acquisition survives an in-flight unlock. |
||
|
Chris Lu
|
6f14be1138 |
stats: remote-mount bucket cache hit/miss metrics (#10352)
Reads of remote-backed entries now record hit or miss in
SeaweedFS_remote_cache_read_total{source,bucket,result} on the filer HTTP
path and the S3 gateway, so cache effectiveness of mounted buckets can be
graphed. Inline-content entries count as hits since they are served
locally without chunks. The filer purges the per-bucket series when the
bucket directory is deleted, so a standalone filer does not accumulate
series across bucket delete/recreate churn.
|
||
|
Chris Lu
|
1fda7aa7f1 |
master: assign re-picks once after its growth concludes instead of shedding (#10348)
The initiator's shed check (initiatedGrow != HasGrowRequest) compares against an err from a PickForWrite that may predate the growth concluding: the grower registers its volumes before clearing the flag, so when growth lands between the failed pick and the check, the assign shed ResourceExhausted even though a writable volume was already registered. Re-pick once after observing the conclusion and shed only if the volume layout still has nothing writable. Applies to both the gRPC Assign and the HTTP dirAssign paths, which share the shed logic. Flaked in CI as TestAssignInitiatorWaitsForItsOwnGrowth; reproduced deterministically by widening the enqueue-to-check window. |
||
 7y-9
|
8bff3b3213 |
fix(volume): reject overflowing needle ID deltas (#10342)
* fix: reject overflowing needle ID deltas Problem: Parsing a file ID with a delta can wrap a valid maximum needle ID back to zero without returning an error. Root cause: Needle.ParsePath added the parsed uint64 delta without checking whether the sum exceeded the needle ID range. Fix: Compare the delta with the remaining uint64 capacity before addition and return a contextual overflow error when it does not fit. Validation: go test ./weed/storage/needle -run ^TestNeedleParsePathRejectsDeltaOverflow$ -count=1; go test ./weed/storage/needle -count=1; git diff --check 10cdaf381875492a2c752d1038797e96ff18208f..HEAD Co-authored-by: Codex <noreply@openai.com> * fix: propagate needle ID delta parse errors Co-authored-by: Codex <noreply@openai.com> * print the needle id in hex in the delta overflow error * batch delete: keep processing after a cookie mismatch * rust volume: reject overflowing needle id deltas --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
c015cc3939 |
generate vtproto marshalers for filer_pb and use them on the metadata log path (#10337)
* generate vtproto marshalers for filer_pb and use them on the metadata log path Reflection-based proto.Unmarshal allocates a fresh message tree through reflect.New on every call. On the metadata subscription fan-out the same event is decoded once per subscriber, so reflect.New tops the decode churn under many mounts. Generate MarshalVT/UnmarshalVT/SizeVT for filer.proto (a separate filer_vtproto.pb.go, filer.pb.go untouched) and call them on the log entry marshal and the subscribe/replay decode paths. UnmarshalVT allocates message structs directly and copies byte and string fields, so it stays wire-compatible with proto.Unmarshal and preserves the non-aliasing the persisted-log cache depends on. For SubscribeMetadataResponse this cuts decode allocations 69 -> 50 and ~4.5us -> ~2.1us per event; the win scales with subscriber overlap. * marshal log entries directly into the buffer SizeVT is allocation-free and MarshalToSizedBufferVT writes into a pre-sized slice, so the log entry can be marshaled straight into logBuffer.buf. This drops the per-entry MarshalVT allocation and the follow-up copy on the write path. * expand vtproto benchmarks: marshal, decode, and marshal-into-buffer by chunk count Parametrize by nested-message count (chunks per event) and add encode + zero-alloc marshal-into-buffer benchmarks alongside the decode one, so the write-path win from MarshalToSizedBufferVT is measurable too. * keep proto.Unmarshal for metadata events to preserve UTF-8 validation UnmarshalVT skips proto3's UTF-8 validation of string fields, so a SubscribeMetadataResponse with an invalid-UTF-8 string (e.g. Directory "\xff") that proto.Unmarshal rejects would decode and reach path filtering and subscribers. Decode events with proto.Unmarshal again; UnmarshalVT stays on the log entry paths, whose only variable-length fields are bytes and so carry no UTF-8 constraint. Tests cover the codec difference and that a malformed event is skipped before delivery. |
||
|
Chris Lu
|
19dc085e33 |
master: statistics used size covers all collections and layouts (#10319)
StatFs on a mount reported cluster-wide total capacity but used size from a single volume layout keyed by collection, replication, ttl, and disk type. A mount without -collection therefore showed only the default collection's usage, hiding data in named collections, and even a collection-scoped mount missed volumes with a different replication, ttl, or disk type. Aggregate used size and file count across all layouts of the requested collection, and across every collection when the collection is empty, matching how Topology.Lookup treats an empty collection. Looking up stats no longer creates a phantom collection as a side effect. |
||
|
Chris Lu
|
fa549e9c83 |
filer: harden TUS session authorization against cross-prefix access (#10315)
* filer: authorize TUS existing-session verbs against the validated stored target Scope-check on TUS HEAD/PATCH/DELETE only populated a resource path for POST, so a prefix-restricted token that learned another tenant's session id could act on that session and land content at a TargetPath its own AllowedPrefixes forbid. Split the filer JWT check into authenticateFilerJwt (signature and method) and authorizeFilerJwtPaths (resource scope), and make the scope check fail closed: a prefix-restricted token with no resolved resource path is denied instead of authorized on signature alone. The TUS handler now authenticates first, reads and validates the session once, authorizes the stored TargetPath, then operates on that single pinned snapshot. readTusSessionInfo rejects a session whose id, target or size is unusable, and getTusSession is split so the authorization lookup no longer lists chunks. * filer: reject non-canonical TUS upload ids The uploads route took the first path component as the session id, so a trailing path or other non-canonical spelling aliased one session under several URLs. Require the id to be a canonical UUID, the only form the server mints, both at routing and when reading a session's metadata, so one URL maps to one resource. * filer: revalidate the pinned TUS session before completing an upload Completion re-read chunks but not the session identity, so a PATCH finishing after a concurrent DELETE or metadata replacement could still land at the id's stored path. Before completing, confirm the session still exists and its target, size and creation time are unchanged from the authorized snapshot; otherwise the completion fails instead of writing to a path the request never authorized. * filer: log TUS session lookup failures before returning not-found readTusSessionInfo and loadTusSessionChunks failures answered "not found" with no log line, so a transient filer or listing error was indistinguishable from a genuinely missing session. Log the lookup at V(1) (a missing session is common and benign) and the chunk-load error at Errorf (the session already resolved). |
||
|
Chris Lu
|
c1a1e3c1e3 |
shell: volume.tier.upload keeps volume replicas (#10314)
* volume: copying a remote-backed volume only needs space for the index VolumeCopy sized its target-location check by the source .dat even when that .dat lives in a cloud tier and only .idx/.vif land locally, so re-replicating a tiered volume demanded the full remote size in free disk. Require the index size instead. * shell: volume.tier.upload keeps volume replicas Tiering a replicated volume deleted every replica but the upload source, leaving one server holding the only .idx and the only .vif that knows the remote object key — losing that server orphaned the volume even though its data sat intact in the cloud. Replicate the uploaded .idx/.vif onto the other replica servers instead (VolumeCopy skips the .dat for remote-backed volumes), so all replicas serve reads from the same remote object and the volume keeps its replica count. An already-tiered replica is preferred as the upload source, so a rerun after a partial failure reuses the existing remote object instead of uploading a second copy under a new key. * shell: group tier upload locations instead of re-prepending * rust volume: copying a remote-backed volume only needs space for the index Mirror the Go VolumeCopy change: size the free-location check by the source .idx when the .dat lives in a cloud tier, since only .idx/.vif land locally. |
||
 Eliah Rusin
|
c006dc563e |
ec: remove .ecsum sidecars on destroy / shard delete; align Go and Rust cleanup (#10307)
* fix(rust-volume): remove .ecsum sidecars on EC destroy / shard delete Rust EcVolume::destroy removed shards and .ecx/.ecj/.vif but left bitrot checksum sidecars (.ecsum / .ecsum.v*). On clusters that run weed-volume (not Go weed volume), collection.delete therefore orphans every sidecar while correctly wiping shards — observed live on 4.39 (14/14 .ecsum survived after collection.delete on a freshly encoded EC volume). Go Destroy already calls RemoveBitrotSidecars; this brings Rust to parity: - hoist remove_bitrot_sidecars into ec_bitrot (shared helper) - call it from EcVolume::destroy for dir / dir_idx / ecx_actual_dir - call it from Store::delete_ec_shards when a disk has no remaining shards - unit test: test_destroy_removes_bitrot_sidecar * rust volume: gate the shard-delete sidecar sweep on a local shard removal Only sweep a disk's .ecsum when this delete actually removed a shard file there, matching Go's found gate: a delete that never touched a disk must not strip a sidecar it does not own — a shared -dir.idx sibling with surviving shards, or an ec.rebuild index-prep copy that lands .ecx/.ecsum before any shard. The shard-presence probe now treats unexpected stat errors as "exists" so a transient failure cannot orphan-classify live shards, and check_all_ec_shards_deleted reuses it. * rust volume: destroy() sidecar sweep needs only the data and idx bases ecx_actual_dir is always one of the two, so the third branch could never run; this is now exactly Go Destroy()'s two-base sweep. * rust volume: call the shared sidecar removal helper directly * rust volume: unit-test remove_bitrot_sidecars Mirrors Go's TestRemoveBitrotSidecars: legacy and versioned sidecars are removed, a shard file and a longer-vid sidecar survive, absent is success. * rust volume: keep the shared idx-base sidecar while a sibling disk has shards One -dir.idx serves every location, so emptying one disk must not sweep <idx>/<vol>.ecsum out from under a sibling that still holds shards. Nothing reads the idx-base sidecar today, but .ecx shows index-dir files are real; this keeps the defensive sweep safe if a writer ever lands one there. * ec shard delete: keep the shared idx-base sidecar while a sibling disk has shards One -dir.idx serves every disk, so emptying one disk must not sweep <idx>/<vol>.ecsum out from under a sibling that still holds shards of the volume — the same gate the Rust volume server applies. A status error counts as in-use so a transient failure never strips it early. * rust volume: drop a shard-only disk's stale .vif with the node's last shard Go's removeEcSharedIndexFiles also clears the data-base .vif in the all-shards-gone pass, gated on .idx absence so a disk still hosting the source volume keeps its live .vif; the Rust delete path left it behind. Unexpected stat errors count as .idx-present so a transient failure never strips a live volume's .vif. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
ce82e3a057 |
filer: scope TUS HEAD/PATCH/DELETE against the session target path (#10309)
* filer: scope TUS HEAD/PATCH/DELETE against the session target path checkTusJwtAuthorization only populated the scoped-path list for POST, so a prefix-restricted token was scope-checked at session creation but not on HEAD/PATCH/DELETE, which act on an existing session addressed by id. A low-privilege tenant holding a valid write token who learns another upload's session id could PATCH attacker bytes into that session, or DELETE/HEAD it, landing content at a target path its own AllowedPrefixes forbids. The session id is unguessable to an unauthenticated attacker but is not an authorization boundary against a legitimate tenant. Resolve the effective target for HEAD/PATCH/DELETE from the session's stored TargetPath and scope the prefix check against it, the same way POST scopes the create target. The scoped paths are resolved lazily so the session read runs only for a prefix-restricted token; unrestricted tokens and deployments without a signing key touch no extra state. An unknown session stays unscoped so the handler still answers 404. * filer: fail closed when a TUS session target cannot be resolved The lazy scope resolver swallowed every readTusSessionInfo error and left the request unscoped, so a filer read error or a corrupt session .info would authorize a prefix-restricted token against a target the server never resolved. Only a genuinely missing session should stay unscoped (so the handler answers 404); any other failure now propagates and denies the request. checkJwtAuthorizationScoped's resolver returns an error and a failed resolution is treated as not-authorized. * filer: fold the scoped-path resolver into checkJwtAuthorization checkJwtAuthorization now takes the lazy resolver directly instead of a separate checkJwtAuthorizationScoped wrapper, and the surrounding comments are trimmed to the load-bearing why. |
||
|
Chris Lu
|
a9cfbd8d3a |
s3: tear down the emptied .versions directory on last-version delete; drain existing residue (#10278)
* s3: routed last-version delete removes the emptied .versions directory The routed versioned delete (routedDeleteSpecificVersion) repoints the latest pointer and deletes the version file, but unlike the lock-path fallback (updateLatestVersionAfterDeletion) it never tears down the .versions/ directory it just emptied. The residue keeps the key's read path in the self-heal rescan loop: every GET of the deleted key logs event=surfaced plus a GetObject error until the background EmptyFolderCleaner gets to the directory — at least two minutes away on its delay queue, and possibly never (the queue is in-memory, bounded, and gated on the bucket's allow-empty-folders policy). Veeam's lock arbitration probes deleted lock keys continuously, so those windows are always open and the log spam is chronic. ObjectMutation DELETE gains remove_empty_parent: after the child delete, the filer best-effort removes the parent directory in the same locked transaction. Non-recursive on purpose — a concurrent write that lands a new version fails the removal instead of being lost with it. The routed last-version delete sets it on the version-file DELETE, matching the lock-path fallback's contract. Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv * s3: drain empty .versions residue on read heal and in s3.versions.audit Directories already stranded by pre-teardown deletes (or dropped from the EmptyFolderCleaner's bounded in-memory queue) previously re-entered the self-heal rescan on every GET forever: the heal only cleared the pointer and nothing ever removed the directory, and s3.versions.audit counted the state as clean. When the heal rescan finds no remaining version, remove the directory outright (non-recursive, so orphan children still block and fall back to the pointer clear) and log event=healed mode=empty_dir_removed; the next GET takes the clean not-found path. The audit gains an empty category so the residue is visible, and -heal removes such directories in bulk. Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv |
||
|
Chris Lu
|
254c2a1024 |
volume: clear remote flag when tiering a volume back to local (#10262)
VolumeTierMoveDatFromRemote downloads the .dat, trims the .vif, and swaps the data backend to the local file, but left hasRemoteFile set. The volume.tier.download command masks this by unmounting and remounting right after, which reloads the flag from the trimmed .vif — but in the window before the remount the in-memory flag is wrong: doDeleteRequest would skip appending the tombstone to the freshly local .dat, and the phantom-.dat guard stays disabled. Give SwapDataBackend a hasRemoteFile argument so the backend swap and the flag move together under one lock, and route both tier directions through it: the tier-down download passes false, LoadRemoteFile passes true. The flag can no longer disagree with the live backend. |
||
|
Chris Lu
|
4f1f0dcb17 |
filer: require JWT authorization on TUS upload endpoints (#10249)
* filer: require JWT authorization on TUS upload endpoints filerHandler and readonlyFilerHandler run every request through maybeCheckJwtAuthorization, but the TUS routes were registered only behind filerGuard.WhiteList, which is a pass-through for the filer (its guard is built with an empty whitelist), and no TUS handler called the JWT check. So with jwt.filer_signing.key set, normal PUT/POST/DELETE were authorized while the TUS endpoints were not. Run the same check in tusHandler before routing: HEAD uses the read key, POST/PATCH/DELETE use the write key, and creation scopes a prefix-restricted token against the resolved target path. OPTIONS stays open for capability discovery. * filer: enforce read-only and WORM rules on TUS completion completeTusUpload wrote the final entry with CreateEntry directly, skipping the read-only and WORM checks the normal write path applies. Reject completion when the target prefix is read-only or the existing entry at the target is WORM-enforced. * filer: align TUS write path with the normal write path Resolve the create target with a guaranteed leading slash so a prefix-restricted token and the stored path stay absolute even if TusBasePath were set with a trailing slash. Reject read-only prefixes at session creation before any chunk is written, and map read-only and WORM rejections at completion to 507 and 403 instead of a generic 500. * filer: cover method-restricted TUS tokens in the auth test |
||
|
Chris Lu
|
a6effe3cfb |
master: repair the lookup index after a vacuum that raced a disconnect (#10226)
* master: re-create a vacuum-committed volume the index has lost SetVolumeAvailable dereferenced vid2location[vid] with no nil check. A disconnect during a long vacuum can drop a single-replica volume from the lookup index while it stays on the node; the commit then panics the master instead of re-registering it. Re-create the entry, and seed its size tracking so assigns are counted right away rather than after the next heartbeat. * master: re-register a vacuumed volume the index lost on mark-writable The maintenance worker re-enables a volume by marking it writable after the vacuum commit, but VolumeMarkReadonly only updated nodes the lookup index already knew. If a disconnect race dropped the volume from the index during the vacuum, that path was a no-op and the volume stayed "not found" until the next full heartbeat healed it. Re-register it from the node that still holds it. |
||
|
Chris Lu
|
6206f60032 |
fix(master): let the growth initiator wait instead of shedding itself (#10202)
* fix(master): let the growth initiator wait for the growth it triggered The growth-in-flight shed also fired on the request that initiated the growth: it sets the pending flag right before the shed check, so a cold-start assign enqueued growth and immediately failed itself with "volume growth in progress". With no concurrent assigns around to pick up the freshly grown volume, a single writer against an empty cluster never completes a write despite ample free space. Claim the pending flag with a compare-and-swap so exactly one request becomes the initiator, triggering growth at most once, and let it wait for that growth to land. Everyone else still sheds retryably instead of pinning a goroutine: followers behind an in-flight growth, an initiator whose growth concluded without yielding a writable volume, and an initiator whose growth outlives the 10s wait budget, which previously surfaced a non-retryable error (gRPC Unknown, HTTP 406) even though a retry would have succeeded moments later. * fix(master): stop assign waits when the request is cancelled The assign retry loops slept through client cancellation, keeping a goroutine spinning for the rest of the 10s budget after the caller had gone; StreamAssign also ran assigns on a background context detached from the stream. Wait on the request context and pass the stream context through. * topology: drop the unconditional grow-request setter Growth is only claimed through AddGrowRequestIfAbsent's compare-and-swap now; keeping the raw Store(true) around invites the check-then-set race back. * test: cover cold-start first write with a real cluster Boot a fresh master plus three empty volume servers and require the very first assign - HTTP and gRPC, each on a cold volume layout, no client retries - to complete a write. The assign that triggers volume growth must wait for it rather than answering "volume growth in progress"; unit tests stub the topology, so only a real cluster exercises the assign-grow-wait path end to end. |
||
 Konstantin Lebedev
|
292abfae33 |
[filer] applyStorageDefaultsToEntry before CreateEntry (#10196)
* applyStorageDefaultsToEntry befor CreateEntry * Update weed/server/filer_grpc_server.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update weed/server/filer_grpc_server.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * add tests * fix: tests * enforce read-only storage rule regardless of explicit TTL, match CreateEntry remote handling * CreateEntry shares applyStorageDefaultsToEntry * routed PUT enforces the path rule's max file name length --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
05b4b5bf56 |
ec: expose force_deleted_needles_check in ScrubEcVolume RPC and shell (#10176)
* ec: expose force_deleted_needles_check in ScrubEcVolume RPC and shell FULL EC scrubs can opt into strict deleted-needle verification via the -forceDeletedNeedlesCheck shell flag, off by default since it can report false positives when EC indexes disagree. Rejected for non-FULL modes. The Rust volume server parses the new field and ignores it: its FULL scrub verifies shards via RS parity, not per-needle reads. * volume: require admin auth for ScrubEcVolume ScrubEcVolume ran unauthenticated while its sibling ScrubVolume, and the rest of the mutating volume handlers, gate on checkGrpcAdminAuth. Close the gap so an EC scrub can't be triggered anonymously. * shell: reject ec.scrub -forceDeletedNeedlesCheck outside full mode Fail in the client before fanning out to every volume server, instead of erroring halfway through once the servers reject the request. |
||
 MorezMartin
|
bdcc3154ed |
refactor: centralize genUploadUrl in UploadOption (#10164)
* refactor: centralize genUploadUrl in UploadOption Replace inline genFileUrlFn closures with operation.GenUploadUrl field: - Add GenUploadUrl func(host, fileId) string to UploadOption struct - Add GenUploadUrlProxy(filerAddress string) utility function - Remove genFileUrlFn parameter from UploadWithRetry signature - Update all callers: mount, gateway, mq, filer_copy, filer_sync This matches the weed mount -filerProxy pattern exactly, factorizing the URL generation logic across all consumers. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * docker release: run all platform jobs in one wave, cache rocksdb compile Drop max-parallel so the 13 per-platform builds run together instead of two waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late). Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is sha-independent, so it caches across releases and stops being the ~16-min long-pole that gates the merge fan-in. go-build variants stay mode=min. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * refactor: centralize genUploadUrl in UploadOption Replace inline genFileUrlFn closures with operation.GenUploadUrl field: - Add GenUploadUrl func(host, fileId) string to UploadOption struct - Add GenUploadUrlProxy(filerAddress string) utility function - Remove genFileUrlFn parameter from UploadWithRetry signature - Update all callers: mount, gateway, mq, filer_copy, filer_sync This matches the weed mount -filerProxy pattern exactly, factorizing the URL generation logic across all consumers. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * Remove accidental ROCmFPX submodule reference * gofmt chunk upload option block * Preserve broker cipher and re-read proxy filer per upload attempt Chunk uploads must keep the configured Cipher, and both the mount and broker current filer can change on failover, so build the proxy upload URL inside the closure instead of capturing the address once. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
77bf2a3ab0 |
volume.balance: gate on real physical disk usage (fixes #10160) (#10162)
* shell: add volume.balance -byDiskUsage to balance by actual data The default balancer ranks servers by slot density, dividing used volumes by MaxVolumeCount. When MaxVolumeCount is configured higher than the disk can hold, a physically near-full server looks nearly empty and gets picked as the move target, so balancing drains less-full servers onto an already-full one. -byDiskUsage ranks servers by the actual data they hold (sum of volume sizes) instead, so the fullest-by-data server is treated as full and balancing drains it. It assumes comparable disk sizes per disk type and still respects each server's free volume slots. Default behavior is unchanged. * plumb physical disk usage into topology, gate volume.balance on it Volume servers now report each disk's filesystem total/free bytes in the heartbeat, and the master stores them in DiskInfo. volume.balance uses them to skip any move target whose disk is already near full (-maxDiskUsagePercent, default 90), so an over-configured maxVolumeCount can no longer make a physically full server look empty and get drained onto. The gate judges each server against its own disk, so heterogeneous disk sizes are fine; servers that do not report bytes fall back to slot-only behavior. Rust seaweed-volume mirrors the heartbeat reporting. * admin: report real physical disk capacity when volume servers provide it The dashboard estimated server capacity as maxVolumeCount * volumeSizeLimit, which overstates it when maxVolumeCount is set higher than the disk holds. Prefer the filesystem capacity now reported per disk, falling back to the estimate for servers that do not report it. * worker: gate automatic balance on physical disk fullness too The maintenance balance worker selects the least slot-utilized server as the move destination, so an over-configured maxVolumeCount makes a physically full server look empty and get drained onto — the same defect as the shell command. Now that DiskInfo carries real disk bytes, skip any destination whose disk is at/above 90% used (per server, against its own disk); a full server can still be a source. When every candidate destination is full, create no tasks. Servers that do not report disk bytes are not gated. * balance: share the physical-disk-fullness gate between shell and worker The shell volume.balance command and the maintenance balance worker each grew their own copy of the disk-fullness gate (targetDiskTooFull / destinationDiskTooFull) and a maxDiskUsagePercent=90 constant. Pull both into weed/topology/balancer (DiskTooFullAfter + DefaultMaxDiskUsagePercent) so the policy has one home and the two balancers can't drift. * balance: harden the physical-disk gate Guard against a nil DiskInfo in the byte/slot lookups. Let a zero disk-capacity report clear previously stored bytes (0 means "not reported" for bytes, unlike maxVolumeCount), so a server that stops reporting falls back to slot-only instead of trusting stale capacity. In the worker, charge each planned move's bytes to its destination within a detection cycle so the gate sees a target fill up rather than only its heartbeat-time free space. Note the per-location capacity summing assumes one location per filesystem (the used ratio the gate relies on stays correct regardless; absolute capacity can over-report). |
||
 Aleksey
|
424cd164e9 |
s3: invalidate stale reader cache locations on chunk read failure (#10156)
* s3: invalidate stale reader cache locations on chunk read failure * filer: share the chunk-read self-heal across reader cache and streaming paths The reader cache retry added a third copy of the invalidate-relookup-compare-retry dance already inlined in PrepareStreamContentWithThrottler and duplicated in retryWithCacheInvalidation. Extract retryFetchWithFreshLocations and route all three through it, parameterized by the refetch primitive. * filer: drop redundant completedTimeNew store in reader cache success path startCaching already stamps completedTimeNew unconditionally before the fetchErr branch; the second store inside the success branch is dead. * filer: make NewReaderCache cache invalidator an explicit parameter The variadic ...CacheInvalidator only ever read the first element, so a caller could pass two and silently get one. Take a single explicit argument and have the non-S3 callers pass nil. * filer: inject reader cache chunk fetch as a struct field Replace the process-global readerCacheFetchChunkData test seam with a per-instance fetchChunkDataFn field defaulted in NewReaderCache, matching how lookupFileIdFn is already wired. Tests set the field on the cache instead of swapping a shared global. * filer: log the location count, not full URLs, on self-heal retry --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
 Lisandro Pin
|
cac83bb4a8 |
Fix scrubbing of deleted needles on EC volumes. (#10130)
EC volumes do not propagate deletions to all shard indexes, so it is possible to run scrubbing on a volume where a deleted needle is still present in the index, or a needle deleted from the index is still present on the volume. On either scenario, scrubbing will fail due to size mismatch errors. This PR reworks the scrubbing logic so needle size mismatches are ignored in such scenarios. Scrubbing can still be forced to check deleted needles (f.ex. to discover index inconsistencies); this option will be exposed in RPCs and `weed shell` on a follow-up PR. |
||
 Rushikesh Deshpande
|
d0db94c34a |
feat(metrics): Add EC rebuild/reconstruct Prometheus metrics (#10124)
* Review comment removed unnecessary success and failure count * fix: use Gather.Gather() with seeded counter for EC rebuild registration test - Restore Gather.Gather() to verify MustRegister calls as requested in review - Seed VolumeServerECRebuildCounter before gathering because CounterVec only appears after at least one label value is observed - Use correct fully-qualified metric names (SeaweedFS_volumeServer_*) * fix: remove preflight checkEcVolumeStatus failure from ec_rebuild_total counter ec_rebuild_total should only reflect actual rebuild execution failures (from RebuildEcFiles / RebuildEcxFile), not scan/precheck failures in the volume status loop. The error is still returned to the caller; only the misleading counter increment was removed. * Review comment removed unnecessary observe * label EC rebuild duration histogram by result Without a result label, fast failures pull down the success-latency quantiles shown on the EC Rebuild Duration panel. Make the histogram a HistogramVec keyed by result, record success/failure through one recordEcRebuild helper, and split the Grafana quantiles by (le, result). * reset EC rebuild metric vecs in registration test The HistogramVec needs a child before Gather emits it, so the test must observe once; reset both vecs in cleanup so that sample doesn't leak into other tests. --------- Co-authored-by: Ubuntu User <ubuntu@example.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
f643893891 |
fix(master): shed assign load when volume growth is already in flight (#10121)
Under a herd of concurrent assigns with no writable volume, Assign spun PickForWrite for the full 10s timeout, pinning a goroutine per request and starving the master of the cycles it needs to process growth and answer heartbeats. When growth is the relevant remedy and already in flight, stop spinning: if free space exists, shed with a fast retryable error so clients back off and retry once growth lands; if the cluster is out of space, fail fast with the real out-of-space error instead of masking it as retryable. The gRPC shed uses ResourceExhausted, not Unavailable: operation.Assign retries it, but the client connection layer doesn't treat it as a dead channel, so a per-request shed across a herd doesn't tear down the shared master connection and cancel every other in-flight assign. The HTTP dirAssignHandler sheds with 503 + Retry-After. |
||
 jk2lx
|
81ed379884 |
volume server: route VolumeMarkReadonly to raft leader (#10120)
* volume server: route VolumeMarkReadonly to raft leader After a master raft election, volume servers may still heartbeat a follower while admin paths such as weed shell volume.mark call notifyMasterVolumeReadonly via vs.GetMaster(). Followers reject VolumeMarkReadonly with NotLeader, which breaks tiering and other mark-readonly workflows until the heartbeat loop reconnects. Resolve the leader through GetMasterConfiguration on configured -master peers (same Leader field filer/master clients already use) before calling VolumeMarkReadonly. When the leader differs from the heartbeat peer, update currentMaster so the heartbeat loop converges faster. Adds operation.LookupRaftLeaderMaster with unit tests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address review feedback on volume.mark raft leader routing Do not update currentMaster during leader lookup — heartbeat owns that field and uses stream GetLeader() to reconnect. Try the heartbeat peer first and only resolve the raft leader after a NotLeader rejection. Add ctx.Err() early exit and quieter logging for context cancellation. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(operation): thread the lookup timeout ctx into connection invalidation The 5s timeout drove only the RPC; WithMasterServerClient saw the unbounded outer ctx, so a self-inflicted timeout (slow GetMasterConfiguration during an election) was treated as a stale channel and tore down the shared master connection. Pass the timeout ctx into the helper so its own expiry leaves ctx.Err() set and spares the connection. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
2efc0e1656 |
ec: recover EC shards whose .ecx index lives only on a peer server (#10108)
* ec: recover EC shards whose .ecx index lives only on a peer server A volume server that boots with EC shard files on disk but no .ecx index on any local disk cannot mount the shards, so the master never learns about them. ec.rebuild works off master-registered shards, so it sees the volume as short and gives up even though the shard data is intact. Add an operator-triggered recovery: VolumeEcShardsMount gains a recover_missing_index flag that makes the volume server fetch the missing .ecx (plus .ecj/.vif) from a peer holding it and mount the on-disk shards. ec.rebuild runs this across the cluster before planning, so orphaned shards register and the rebuild sees the true shard set. .ecx is an immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal that differs across holders, so the recovered node adopts the source peer's deletion view, like a balanced or rebuilt shard does. * ec: mirror missing-index recovery into the Rust volume server Port the #10104 recovery to seaweed-volume so the Rust volume server self-heals the same layout: EC shards on disk with the .ecx index only on a peer. Adds collect_ec_volumes_missing_index / mount_recovered_ec_shards to the store, recover_missing_ec_indexes (master LookupEcVolume + peer CopyFile fetch + mount) to the server, and the recover_missing_index flag on VolumeEcShardsMount. .ecx is the immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal, so the recovered node adopts the source peer's deletion view, matching the Go path. |
||
 sshhan
|
a1fff50935 |
fix(postgres): prevent uint32 underflow & OOM in message parsing (#10099)
* fix(postgres): prevent uint32 underflow & OOM in message parsing * postgres: drop redundant startup guard, use maxStartupMessageSize const The msgTotalLen < 8 check already guarantees msgLength >= 4, so the extra msgLength < 4 guard before reading the protocol version was unreachable. Point the startup size limit at maxStartupMessageSize instead of a literal. * postgres: trim query terminator safely, cap pre-auth payloads Use strings.TrimSuffix for the simple-query null terminator so a non-null-terminated body isn't silently shortened, matching the auth handlers. Bound password/MD5 reads with a dedicated maxAuthMessageSize (10 KiB) instead of the 100 MiB maxMessageSize, since these payloads are read before authentication. --------- Co-authored-by: shangshuhan <shangshuhan@cmict.chinamobile.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
95427b5573 |
security: add BearerPrefix constant for Authorization headers (#10101)
Introduce security.BearerPrefix ("Bearer ", RFC 6750) and use it
everywhere an "Authorization: Bearer <token>" header is constructed,
replacing the scattered "BEARER "/"Bearer " string literals. SeaweedFS
matches the scheme case-insensitively when parsing (security.GetJwt), so
behavior is unchanged; this removes the magic string and settles the
casing on the standard form. The parser's upper-case comparison stays as
is on purpose.
|
||
|
Chris Lu
|
4d3e5d94a9 |
filer: mint volume read JWT when proxying chunk reads (#10100)
The /?proxyChunkId= endpoint forwards the caller's headers to the volume server but never mints a read token, so proxied chunk reads return 401 once jwt.signing.read.key is configured. Generate a fileId-scoped volume token the same way the direct filer read path does, which fixes filer.sync, filer.backup, filerProxy mounts, the MQ broker and the upload gateway in one place. |
||
|
Chris Lu
|
96d2d13efe |
s3: replicate by fanning out from the gateway to every holder (#10078)
* s3: replicate by fanning out from the gateway to every holder The S3 gateway uploaded each chunk to one volume server, which then relayed the copies to the other replica holders. The gateway now uploads each chunk to every holder in parallel (type=replicate), removing the primary volume server's receive-then-resend relay. AssignVolume returns every replica holder (new repeated Location replicas, forwarded from the master assign), the s3api captures them, and the chunked uploader fans out whenever a chunk has more than one holder. Cipher uploads keep the server-driven path since per-call encryption would diverge the replicas. * s3: cancel sibling replica uploads on the first failure * s3: trim replica fan-out comments * s3: roll back successful fan-out chunk copies when a holder fails A failed fan-out records no FileChunk, so copies that landed on the holders that finished before the cancel were leaked as orphans the caller could not see. Track the holders that succeeded and delete the needle from each (type=replicate, local-only) on failure, leaving nothing behind. |
||
|
jk2lx
|
e1f89f85f2 |
fix(filer): apply -filer.disk default to metadata log assigns (#10080)
* fix(filer): apply -filer.disk default to metadata log assigns Metadata event log writes call operation.Assign directly and used only FilerConf path rule DiskType. When filer.conf rules were missing or unmatched, the master received an empty DiskType and grew volumes on the built-in hdd layout. Mirror resolveAssignStorageOption: wire FilerOption.DiskType into the Filer, fall back when the matched path rule has no disk type, and return the matched rule from resolveMetadataLogAssignDiskType to avoid duplicate MatchStorageRule lookups. Co-authored-by: Cursor <cursoragent@cursor.com> * mini: fall back to -volume.disk for filer default disk type weed server copies -volume.disk into the filer disk default when -filer.disk is unset; weed mini did not, so metadata-log assigns sent an empty disk type on clusters that only tag volumes (e.g. hot/warm). --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
 qzhello
|
fb168e2a36 |
fix: avoid reading upload body when writing JSON errors (#10073)
* fix(shell): correct volume.list -writable filter unit and comparison * fix(shell): correct volume.list -writable filter unit and comparison * chore(shell): fix typo in EC shard helper param names * fix(shell): use exact match for volume.balance -racks/-nodes filter The old strings.Contains-based filter quietly included any id that was a substring of the user-supplied flag value (e.g. -racks=rack10 also matched rack1). Replace it with an exact-match set parsed from the comma-separated flag value, and add regression tests for both -racks and -nodes paths. Also fix a small typo in the "remote storage" error returned by maybeMoveOneVolume. * fix(shell): use exact match for volume.balance -racks/-nodes filter The old strings.Contains-based filter quietly included any id that was a substring of the user-supplied flag value (e.g. -racks=rack10 also matched rack1). Replace it with an exact-match set parsed from the comma-separated flag value, and add regression tests for both -racks and -nodes paths. Also fix a small typo in the "remote storage" error returned by maybeMoveOneVolume. * refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers * fix: avoid reading upload body when writing JSON errors |
||
 198wmj
|
aeaf62fa86 |
fix: resolve postgres startup message length type mismatch and uint underflow OOM risk (#10065)
* fix: resolve postgres startup message length type mismatch and uint underflow OOM risk * Update weed/server/postgres/server.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: wangmeijuan <542204218@qq.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> |
||
|
MorezMartin
|
6f1d4af035 |
fix(filer): propagate proxyChunkId query params to volume server (#10036)
* fix(filer): propagate proxyChunkId query params to volume server
When weed mount reads via filer proxy mode (-volumeServerAccess=filerProxy),
the mount adds query params like readDeleted=true to chunk read requests.
Two bugs prevented these from working:
1. filer_server_handlers.go extracted fileId from the raw RequestURI, which
includes query params, corrupting the fileId (e.g. '6,abc&readDeleted=true').
Fix: use r.URL.Query().Get("proxyChunkId") for clean extraction.
2. filer_server_handlers_proxy.go didn't forward query params to the volume
server. The urlStrings from LookupFileId already contain the fileId in the
path, so just append the original query string.
* filer: match chunk proxy by query param, not URI prefix order
Order-dependent prefix slicing missed proxyChunkId when it wasn't the
first query param. Gate on root path and read the parsed query value.
* filer: drop internal proxyChunkId from proxied volume query
Lookup URLs already carry the fileId in the path, so forwarding the raw
query duplicated proxyChunkId onto the volume server. Strip it and only
append the remaining caller params (e.g. readDeleted).
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
|
||
 Bruce Zou
|
7688e69146 |
use Leader() instead of MaybeLeader() in SendHeartbeat (#10029)
During leader election, MaybeLeader() returns empty string immediately (non-blocking), causing master to return NotLeaderError without sending HeartbeatResponse.Leader. Volume servers depend on HeartbeatResponse.Leader to discover the new leader address, so they keep retrying the old leader and cannot switch. Switching to Leader() restores the 20-second exponential backoff retry behavior, ensuring volume servers receive the new leader address as soon as election completes. |
||
|
Rushikesh Deshpande
|
df1a25fd3e |
feat: add Prometheus metrics for replication operations (#10006)
* feat: add Prometheus metrics for replication operations Adds 5 metrics to instrument volume server replication (write/delete): - Operations counter with success/failure labels - Duration histogram for latency tracking - Targets gauge for replica fanout - Failures counter with error reason labels - Under-replicated volumes gauge on master * fix: record replication duration histogram only when replicaCount > 0 * fix: update replication targets gauge for all operations including zero * fix: ensure symmetric replication success/failure counting and proper metrics updates * fix: change VolumeServerReplicationTargets from Gauge to Histogram - Replace .Set() with .Observe() in store_replicate.go (2 occurrences) - Update test to use CollectAndCount for histogram assertion - Rename TestReplicationTargetsGauge -> TestReplicationTargetsHistogram - Update documentation to reflect Histogram type and PromQL examples * Add comments to replication metrics and improve test coverage * metrics: add replication panels to grafana dashboard Master row gets an under-replicated volumes timeseries; Volume Servers row gets replication operations, failures-by-reason, p99 duration, and average fan-out panels for the new replication metrics. * metrics: name the replication duration histogram replication_seconds Match the volumeServer convention (request_seconds, vacuuming_seconds) rather than the admin/lifecycle _duration_seconds spelling. * metrics: guard replication fan-out panel against divide-by-zero clamp_min the _count rate so the avg-targets ratio reads 0 instead of NaN when there are no replication events in the window. --------- Co-authored-by: Ubuntu User <ubuntu@example.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
7df43ad9b5 |
admin: add connected Mount Clients page and dashboard section (#9968)
* admin: add connected mount clients page and dashboard section
The filer is the authority on who is subscribed to its metadata stream
(FUSE/VFS mounts, S3, peer filers, ...), but its in-memory listener
registry only tracked clientId->epoch and was not exposed.
- Enrich the filer subscriber registry with name/type/address/path/
connected-time, populated in addClient and cleared in deleteClient so
it reflects currently-connected clients only.
- Add a ListMetadataSubscribers filer gRPC (optional client-type filter).
- Admin server fans out to every filer, filters to mount types
("mount" Go weed mount, "sw-vfs" Rust VFS), and renders a new
Cluster > Mount Clients page plus a Mount Clients dashboard section.
Read-only; no behavior change to the subscribe hot path.
* admin: address review — parallelize filer fan-out, guard nil map, robust CSV
- GetMountClients now queries filers concurrently, each under a 5s
timeout, so a slow/unreachable filer can't stall the admin dashboard.
- Defensively initialize fs.subscribers before first write.
- Mount Clients CSV export uses a Blob with quote-escaping instead of a
data: URI, so special characters in paths export correctly.
|
||
|
Chris Lu
|
a736ba1c21 |
filer: keep metadata-subscription send gauge fresh on idle heartbeat (#9966)
* filer: keep metadata-subscription send gauge fresh on idle heartbeat
last_send_timestamp_of_subscribe only advanced when a real matching
metadata event was streamed to a subscriber. On a quiet path an idle but
perfectly healthy subscriber therefore looked increasingly stale, and the
dashboard panel rendered a large, misleading 'lag'.
An idle heartbeat is a send too, so advance the gauge when one is emitted.
Subscribers that opt into idle heartbeats (filer.sync) now report true
freshness; the rest still show time since the last real event.
Rename the dashboard panel 'Metadata Subscription Lag' ->
'Time Since Last Subscription Send' and clarify its description to match.
* filer: guard nil option when advancing heartbeat gauge
maybeSendIdleHeartbeat is unit-tested with a bare &FilerServer{} (nil
option), so dereferencing fs.option.Host for the sourceFiler label
panicked. Guard it: production always has option set; the test now gets
an empty sourceFiler label instead of a nil-pointer panic.
|
||
|
Chris Lu
|
c7781bfca2 |
fix(ec): remove shared EC index only when no shard remains node-wide (#9955)
* fix(ec): remove the shared EC index only when no shard remains node-wide deleteEcShardIdsForEachLocation removed the shared .ecx/.ecj/.vif index as soon as a single disk's shard count hit 0, even when a sibling disk of the same node still held shards of the volume (split-disk reconciled layout) -- orphaning those shards without their index. Split the non-teardown delete into two passes: delete the requested shard files (and now-orphaned per-disk bitrot sidecars) on every disk, then remove the shared index only once no shard of the volume remains on ANY disk. This brings the Go volume server in line with the Rust one, which already gates the index removal on a node-wide check. * refactor(ec): reuse checkEcVolumeStatus across the two delete passes Address review: cache hasEcxFile/hasIdxFile from the node-wide count pass and pass them to removeEcSharedIndexFiles instead of re-listing each location's directory. * fix(ec): clean an orphaned EC .vif even when its .ecx is already gone Address review: removeEcSharedIndexFiles returned early on !hasEcxFile, so a node-wide teardown left a stale EC .vif behind when its .ecx was already removed. Decouple the .vif removal (gated on !hasIdxFile) from .ecx presence so the generation metadata doesn't leak once no shard remains node-wide. |
||
|
Chris Lu
|
284796c7b6 |
fix(ec): fence stale-worker EC shard cleanup by encode generation (#9953)
* feat(ec): add encode_ts_ns to the EC task params, shard-unmount, and shard-delete RPCs The generation fence for stale EC-worker cleanup needs the encode generation on three messages: ErasureCodingTaskParams (admin issues it), VolumeEcShardsUnmountRequest, and VolumeEcShardsDeleteRequest (the worker carries it to the volume server). Additive fields only; 0 preserves the existing unfenced behavior. Mirror the two volume-server fields in the Rust volume server's proto copy. * feat(ec): issue the EC encode generation from the admin and carry it on the worker Stamp each EC proposal's encode_ts_ns from the admin's per-cycle DetectionSequence (a single-clock value) so generations are globally ordered even though detection runs on a rotating worker. The worker writes that generation into the distributed .vif and passes it on its shard unmount/delete RPCs; it falls back to a local timestamp for the .vif only on the unfenced legacy/shell path (keeping the read guard on). * fix(ec): fence the stale-worker EC shard unmount and teardown by generation A reaped-but-still-running EC worker's cleanupStaleEcShards issued a generation-blind unmount + full teardown that could unmount and then overwrite a newer run's live shards on a shared node. Both RPCs now carry the encode generation: the volume server unmounts/deletes a disk only when its .vif generation is strictly older than the request, and preserves a same-or-newer generation, a generation-0 (recovered or pre-upgrade) volume, and an unreadable .vif. Unload is per-disk, never node-wide. Request generation 0 keeps the blanket teardown for the shell pre-encode cleanup and pre-upgrade callers. Mirrored in the Rust volume server. * test(ec): cover the generation-fenced teardown and unmount End-to-end volume-server tests: a fenced FullTeardown wipes a strictly- older generation, preserves a newer one, preserves a generation-0 volume, and blanket-wipes on request generation 0; the gen-aware unmount preserves a same-or-newer mounted generation; and the .vif generation reader handles present/absent/no-config cases. * test(ec): pin the fenced .vif==teardown generation and the unreadable-.vif preserve A fenced run must stamp the admin generation verbatim into the .vif so it matches the generation sent on the teardown RPCs; add a regression test that sets the task generation and asserts the .vif carries it exactly. Also cover the present-but-unparseable .vif case (reads as generation 0, preserved) and correct the readEcGenerationTsNs docstring accordingly. * fix(ec): surface EC full-teardown filesystem errors in the Rust volume server remove_ec_volume_files(_full_teardown) discarded every fs::remove_file error, so a teardown that failed on permissions or a full disk still returned full_teardown_done=true and left stale artifacts to collide with the next encode. Return io::Result, ignore NotFound, propagate the first real error, and have the teardown RPC surface it -- matching the Go contract. The best-effort reconcile/load-cleanup callers keep ignoring it. * refactor(ec): reuse the EC volume lookup on unmount and short-circuit the gen read Address review: the Rust unmount fence reuses the ec_vol it already fetched instead of a second find_ec_volume; the Go .vif generation reader breaks out of the data/idx loop early when the two dirs are the same. |
||
|
Chris Lu
|
94357ac6a9 |
[volume] preserve compression state during replication (#9946)
* preserve compression state during replication * explain why ParseUpload skips compression for replica writes * fix data race on err result in FetchAndWriteNeedle The local-write and replica-write goroutines all wrote the named err return under an unsynchronized err==nil check. Give each goroutine its own error slot and combine after wg.Wait(): local error wins, then the first replica failure. * skip redundant decompression of compressed needles during replication doUploadData decompressed a compressed input only to report the clear-data length on UploadResult.Size, which both replication callers discard. Skip the decompress when IsReplication. |
||
|
Chris Lu
|
4fb3e22a01 |
fix(tiering): never delete a shared remote object while replicas still reference it (#9942)
* tiering: stop a shared remote object being deleted while replicas still point at it A remote-tiered volume's .dat content lives only in one cloud object that all N replica .vif files point at. Deleting that object while destroying any one replica, or before a downloaded replica is durable, bricks the survivors. - volume.tier.move cleanup now deletes old replicas with keepRemoteData=true so surviving replicas keep the shared object. Document why the alreadyPlaced anchor needs no replica sync (same-object replicas are byte-identical). - VolumeTierMoveDatFromRemote now fsyncs the downloaded .dat, fsyncs the containing directory, trims the .vif (fsynced) and swaps to the local DiskFile BEFORE deleting the remote object, on both the keep-remote and delete paths. Only the final DeleteFile is gated by keep_remote_dat_file, so a keep-remote download leaves the replica served from local disk rather than the shared object, and a crash before delete merely leaks the object. - volume.tier.download keeps the shared object for every replica except the last, which deletes it. - s3 and rclone download paths fsync the .dat before close. * storage: swap the volume data backend under the data lock The tier-download swap closed v.DataBackend and assigned the new local DiskFile without holding dataFileAccessLock, racing concurrent reads/writes (use of a closed file / nil deref). Add an exported Volume.SwapDataBackend that performs the close-and-replace under the lock, and call it from the tier download. * server: skip directory fsync on Windows in the tier download path os.Open(dir).Sync() is unsupported on Windows and returns an error, which would fail VolumeTierMoveDatFromRemote entirely there. Skip the directory fsync on Windows, matching how the storage-side helper tolerates the unsupported case. * shell: make multi-replica tier.download resilient to already-local replicas If a multi-replica download is interrupted and retried, a replica made local in the prior attempt returns "already on local disk", which aborted the whole command and left the remaining remote replicas dangling. Treat that case as a skip-and-continue so a retry completes the rest. * server: assert downloaded .dat content, not just length, in the tier test A length-only check passes even if the bytes are corrupted; compare the full content of the local .dat against the original. |
||
|
Chris Lu
|
c2591b4395 |
fix(replication): verify-before-destroy in VolumeCopy, check.disk, and over-replication trim (#9943)
* volume: verify before destroy in VolumeCopy and replication repair Four data-safety fixes around copy/repair paths that could destroy or resurrect data before verifying the source or survivors. (a) VolumeCopy no longer deletes a pre-existing local replica up front. The delete is deferred until ReadVolumeFileStatus on the source succeeds, so a transient source outage (or a retry after one) can no longer wipe a healthy destination replica. Gated on source readability only; size/count comparisons are intentionally not used because they invert legitimately after divergent vacuum/compaction. Mirrored in the Rust volume server. (b) volume.check.disk no longer resurrects vacuumed-deleted needles. A key present-and-live on the source but entirely absent on the target is ambiguous: it may be a genuine missing write, or a needle deleted on the target and then vacuumed (its index entry and any tombstone are gone). An individual needle AppendAtNs has no monotonic relation to a vacuum watermark, so the old cutoff heuristic could not tell them apart. Without positive proof the absence is a missing write, the safe default is to NOT push it back. Tradeoff: a real missing write may go unrepaired until a tombstone-aware path exists, but we never raise back deleted data. (c) Over-replication trim no longer resurrects needles or removes the wrong replica. The pre-delete sync now runs read-only (divergence check only) instead of writing the doomed replica's needles into the survivor. pickOneReplicaToDelete only ever removes the smallest of multiple healthy writable replicas; it refuses the trim when doing so would leave only read-only/integrity-flagged survivors, since file_count>0 alone cannot prove the survivor's .dat is readable. (d) Incomplete-volume (.note) cleanup keeps the shared .vif when an .ecx for the same vid coexists on the disk, so removing an interrupted regular copy cannot strip a coexisting EC volume's info file. VolumeCopy now surfaces .note write/remove errors instead of ignoring them. In the Rust volume server (where a persisting note is actually reachable) the .note check moves below the empty-stub sweep and EC validation, keeps the .vif on EC coexistence, and the mount path fails when a .note still persists. * shell: scope the over-replication writable-survivor guard to the trim path only The writable-survivor guard (never trim down to a read-only survivor) lived inside the shared pickOneReplicaToDelete, so it also gated the misplaced-volume relocation via pickOneMisplacedVolume -- a misplaced read-only volume (e.g. a full one) would silently stop being rebalanced. Extract pickSmallestReplica for the relocation path (which deletes-and-recreates and must act on read-only replicas), and keep the writable-survivor guard only in pickOneReplicaToDelete used by the over-replication trim. * seaweed-volume: recompute keep_vif after invalid-EC cleanup in the .note path keep_vif used the pre-validation ecx_exists snapshot, so when the EC-validation step above removed the invalid .ecx/shards, the .note cleanup still preserved a now-orphaned .vif. Re-check .ecx existence at cleanup time, matching the Go hasEcxFile re-check. * shell: keep placement when picking an over-replication victim to delete The trim picked the smallest writable replica without regard to placement, so it could delete the only replica in a required failure domain (e.g. with "100" and replicas dc1 + two in dc2, deleting dc1 leaves both survivors in dc2). Prefer a writable replica whose removal still satisfies placement, falling back to the smallest writable only when none does. |
||
|
Chris Lu
|
aabd44fbb5 |
[volume] preserve volume data mtime across tier moves (#9947)
* fix(tier): preserve volume data modification time * fix(tier): best-effort restore of data mtime on download A failed Chtimes should not abort an otherwise complete tier-down; warn and continue, matching the EC copy path. * fix(tier): preserve volume data mtime in rust volume server Mirror the Go fix: store the source .dat mtime on upload instead of the upload time, and restore it on the downloaded .dat. Without this a tiered-then-restored volume loads last_modified_ts_seconds from the upload/download time, extending its TTL across a restart or remount. * fix(tier): read source mtime via DiskFile.GetStat() GetStat() is nil-safe when the backend is closed concurrently and skips a redundant stat syscall; its cached modTime is the on-disk mtime a reload reads, since every .dat write or Chtimes is followed by a DiskFile (re)open. * fix(tier): surface mtime-restore failures on rust tier-down set_file_mtime now returns io::Result; the tier-down path warns on a failed restore instead of dropping it silently, so a wrong local .dat mtime (and the TTL drift it causes) is observable. Matches the Go download. The EC copy path keeps its best-effort silence. |
||
|
Chris Lu
|
0345658ea8 |
[s3] validate indirect filer path inputs (#9931)
* s3: validate indirect filer path inputs * s3: avoid query parsing on common request path * filer: scope copy/move source against JWT AllowedPrefixes maybeCheckJwtAuthorization only checked r.URL.Path, but copy and move read their source from the cp.from / mv.from query params. A prefix-restricted token could copy or move data out of a subtree it cannot otherwise reach. Check every path the request touches, reusing pathHasComponentPrefix so `..` in the source is collapsed before the prefix match. * s3: confine iceberg CreateTable location to the catalog bucket CreateTable derived the metadata bucket and path from the client-supplied req.Location / req.Name and wrote there directly, so a caller scoped to one table bucket could place metadata in another bucket (and path.Join collapsed any `..`). Require the parsed bucket to equal the request's catalog bucket and reject traversal segments in the table path. * webdav: clean client path before subFolder confinement wrappedFs concatenated subFolder + name before the underlying FileSystem ran path.Clean, so `..` in the request path or COPY/MOVE Destination resolved across the FilerRootPath confinement boundary. Clean the name as a rooted path first so traversal segments collapse below subFolder. Only the non-default -filer.path (non-empty subFolder) setup was affected. * filer: enforce read-only rule on real write path with destination header The x-seaweedfs-destination header overrides the path used for storage-rule matching while the entry is written at r.URL.Path, letting a caller select a writable rule for a read-only target. When the header is present, also check the read-only/quota rule against the actual write path. |
||
|
Chris Lu
|
79ac279fe1 |
fix(ec): don't mix EC shards from different encode runs (#9880)
* feat(ec): add encode_ts_ns to EC shard metadata and the shard read RPC EcShardConfig and VolumeEcShardReadRequest gain an int64 encode_ts_ns (encode time in unix nanos). It rides in .vif and the read request so a read can be scoped to the encode run that produced the index. * fix(ec): stamp each encode and reject cross-run shard reads Generate stamps EncodeTsNs into the volume's .vif. Reads carry it to the shard's owning volume (resolved together via FindEcVolumeWithShard, so a multi-disk server validates the disk that actually serves the bytes) and reject a shard from a different encode run, recovering from parity. A zero on either side (pre-upgrade volume) skips the guard. * fix(ec): stamp the encode identity on the worker-generated .vif The worker-local encode path now writes EncodeTsNs (and the resolved EC ratio) into the .vif, so the read guard is not silently off for volumes encoded by the maintenance worker. * fix(ec): wipe stale EC artifacts before re-encoding VolumeEcShardsGenerate evicts any in-memory EcVolume for the volume and removes its on-disk shard/index/sidecar files before writing fresh ones, so a retried encode never builds on a partial prior run and the unlink frees the inodes instead of leaving open fds serving old bytes. * fix(ec): unmount EC shards across all disks UnmountEcShards walked only the first disk holding the shard, leaving a duplicate copy mounted on a sibling disk (split-disk reconciled volumes) still serving and heartbeating. Traverse every disk and emit one deletion delta per disk. * fix(ec): delete orphan shards without a local .ecx deleteEcShardIdsForEachLocation gated shard-file removal on a local .ecx, so it could not clean an orphan .ecNN left by a failed copy on a disk with no index. Delete the requested shard files unconditionally; the index-file (.ecx/.ecj/.vif) routing stays gated as before. * fix(ec): clear stale EC shards cluster-wide before re-encoding ec.encode unmounts and deletes EC shards for the target volumes on every node before regenerating: fatal for the shards the topology reports (mounted leftovers), best-effort for the rest (a sweep that catches unmounted failed-copy orphans). A down node is a no-op. * fix(ec): don't nil EC fds on close so reads can't race eviction A reader resolves an EcVolume/shard under the lock then reads after it is released, so an eviction that nils ecxFile/ecdFile would race that read and panic. Close the fds without nilling the fields: the field is now write-once (no data race) and a concurrent read hits a closed fd, getting a clean error that the caller recovers from parity. * fix(ec): wipe stale EC artifacts on every disk and surface failures The pre-encode wipe only deleted beside the source volume, so a stale shard on a sibling disk survived and could be mounted against the new index at reconcile. Sweep every disk. Removal also ignored os.Remove errors, reporting a failed cleanup as success and letting a stale shard join the next generation; surface the first real failure (treating already-gone as success) from removeStaleEcArtifacts and the shard delete. * fix(ec): log when a local shard is skipped for a different encode run The cross-run guard returned errShardNotLocal, indistinguishable in logs from a genuinely-absent shard. Add a V(1) line naming both EncodeTsNs so operators can tell "wrong encode generation" from "shard not here". * fix(ec): surface metadata removal failures in the shard delete path deleteEcShardIdsForEachLocation still dropped os.Remove errors on the .ecx/.ecj/.vif/sidecar cleanup. A surviving stale .ecx is the orphan-index condition this path prevents, so route those through removeFileIfExists and return the first real failure instead of reporting cleanup as success. * fix(ec): fail orphan cleanup when a reachable node's delete fails The pre-encode orphan sweep swallowed every error for unreported (node, volume) pairs. That is only safe for an unreachable node, which cannot receive this encode's new generation. A reachable node whose delete genuinely failed (permission/IO) keeps an orphan shard that a later copy re-stamps with the new run's volume-level .vif identity, so the read guard would accept stale data. Surface those; stay best-effort only for unreachable nodes (gRPC Unavailable / no status). * fix(ec): guard ecjFile under its lock in the EC delete path EcVolume.Close nils ecjFile under ecjFileAccessLock; a delete that resolved its .ecx lookup before a concurrent eviction (the generate-time UnloadEcVolume) could then reach the journal append with a nil fd. Bail with a clear "volume closed" error under the lock instead. * fix(ec): reject an unstamped shard when the caller has an encode identity The read guard required both identities nonzero, so a current (stamped) caller accepted a holder with identity 0 and could be served a stale pre-upgrade shard. Reject when the caller is stamped and the holder differs (including unstamped); stay lenient only when the caller itself has no identity (pre-upgrade reader). A skipped shard recovers from parity. * fix(ec): full-teardown delete so cluster cleanup wipes a whole generation The pre-encode cluster sweep deleted only the listed canonical shards on remote nodes, leaving index/sidecar (and, on builds with versioned generations, those too) behind. Add a full_teardown flag to VolumeEcShardsDelete that evicts the volume and wipes every EC artifact for it on every disk via removeStaleEcArtifacts; the shell and worker pre-encode cleanup paths set it. Other delete callers (balance/decode/repair) are unchanged. * fix(ec): take ecjFileAccessLock before the nil-check in Sync and Close Sync and Close read ev.ecjFile before acquiring ecjFileAccessLock while Close nils it under the lock, a data race on the field. Take the lock first, then nil-check inside, in both. * fix(ec): acknowledge full_teardown so a pre-upgrade server can't fake success An old volume server silently ignores full_teardown and returns success for an ordinary delete, so the caller wrongly believes the generation was wiped and copies a fresh gen-0 onto an unwiped node. Echo full_teardown_done in the response; the worker destination cleanup fails when it is absent, and the shell cluster sweep fails for a reported (mounted) leftover while staying best-effort for an unreported node. encode_ts_ns stays an accepted transient (an old server just skips the new read guard, no regression). * fix(ec): fail the pre-encode sweep for any reachable node that can't ack teardown A reachable pre-upgrade server ignores full_teardown and returns success without wiping an orphan, which a later copy then folds into the new generation. Treat a missing full_teardown_done ack as fatal for every reachable node (best-effort only for a gRPC-unreachable one), not just for topology-reported pairs. * fix(ec): return the served shard identity and validate it client-side The encode identity was only enforced server-side, so a pre-upgrade server ignored the request field and served bytes unchecked. Echo the served shard's EncodeTsNs on every read response chunk and have the client reject a mismatch (including 0 from an old server), so the guard holds regardless of server version; a rejected read recovers from parity. * fix(ec): reject a short/empty remote shard read instead of serving zeros doReadRemoteEcShardInterval accepted an immediate EOF or a short stream and returned success with a partly zero-filled, unvalidated buffer (the server stamps the identity only on chunks that carry bytes). A non-deleted interval must arrive whole: require n == len(buf), exempting the is_deleted short-circuit (n=0), matching readLocalEcShardInterval's local check. A short read now fails so the caller recovers from parity. * test(ec): fake volume server echoes the full_teardown acknowledgement The worker now fails a teardown delete that isn't acknowledged (so a pre-upgrade server can't silently skip the wipe). The fake server's no-op VolumeEcShardsDelete returned an empty response, which the worker read as a skipped teardown and aborted the encode. Echo full_teardown_done. * feat(ec): mirror the encode-run identity guard + full_teardown into the Rust volume server The Go volume server stamps an encode-run identity (encode_ts_ns) into the .vif and rejects a read served from a shard of a different run; full_teardown wipes a whole generation and acknowledges it. The Rust volume server had none of it. Mirror the shared logic: load encode_ts_ns from the .vif onto the EcVolume, stamp it on every read response, and reject a request/response mismatch on both the server and the distributed-read client (recovering from parity); handle full_teardown by evicting the volume and wiping every EC artifact on each disk, echoing full_teardown_done so the caller can detect a server that ignored it. * fix(ec): remove a stale .vif on full teardown of a shard-only node A shard copy installs shards + .ecx before .vif, so an interrupted copy after a teardown could mount the new files under the previous run's identity / version / shard ratio / dat_file_size carried by the surviving .vif. Remove .vif during full teardown, gated on .idx absence so a source-volume holder keeps its live .vif. In Rust this lives in a teardown-only helper so the reconcile / load- fallback paths (which share the base removal) still preserve .vif. * fix(ec): treat a missing teardown ack as fatal, not as an unreachable node isNodeUnreachable returned true for any non-gRPC-status error, so a reachable pre-upgrade server's missing full_teardown_done ack (a plain error) was classified unreachable and the unreported pair was silently skipped. Classify only a real codes.Unavailable as unreachable, and wrap the missing ack in a sentinel the sweep treats as fatal regardless. A genuinely down node still surfaces as Unavailable from the RPC and stays best-effort. * fix(ec): reject a short shard read in the local EC needle reader read_ec_shard_needle ignored the byte count from shard.read_at and appended the whole pre-sized buffer, so a truncated shard's zero-filled tail passed the later length check and parsed as garbage. Require n == buf.len() per interval, erroring on a short read like the local interval reader already does. * fix(ec): probe reachability before skipping a node that returns Unavailable The pre-encode sweep skipped any node whose teardown delete returned codes.Unavailable, but a reachable volume server in maintenance mode also returns that code for the maintenance-gated delete, so its stale EC files were left behind on a node that can still receive the new generation. Confirm with a non-maintenance-gated empty-target Ping: skip only when the node fails the probe too (genuinely unreachable). * fix(ec): use try_exists for the teardown .vif .idx guard The teardown-only .vif removal gated on Path::exists(), which returns false on a permission/IO stat error, so a stat failure on a present .idx would read as a shard-only node and delete the live source volume's .vif. Gate on try_exists() == Ok(false) instead, preserving the sidecar on any stat error. * fix(ec): only skip a sweep node when a Ping confirms it is transport-down The pre-encode sweep skipped a node whenever its teardown delete and a liveness Ping both failed, but it treated ANY Ping error as down — an application-level Internal/ResourceExhausted, or Unimplemented from a pre-Ping server, left a reachable node's stale generation in place. Classify the Ping tri-state and skip only when it transport-fails with codes.Unavailable; a reachable or inconclusive node stays fatal. * fix(ec): exclude sweep-skipped nodes from the encode's rebalance The pre-encode sweep skips a genuinely-down node best-effort, but the rebalance then recollected the current topology — a node that recovered between the two could become a copy target and receive the new generation while still holding its stale, never-cleared shards. Have the sweep return the skipped set and exclude those nodes from the rebalance for this encode, so a node we could not clean cannot receive the new generation. Standalone ec.balance is unaffected. * fix(ec): re-sweep recovered nodes before generation so they aren't stranded A node skipped as down by the pre-encode sweep is excluded from the rebalance, but it can recover and become the generation host — mounting all shards locally, then being excluded from distribution. Union-only verification accepts all shards on one node and deletes the originals: a single point of failure. Re-sweep the skipped nodes just before generation; one whose teardown now succeeds leaves the skipped set and rebalances normally, while a node still down stays skipped. * fix(ec): abort the encode if a selected source is still skipped after re-sweep The re-sweep un-skips a recovered node, but the source was selected before it and a node can stay down through the re-sweep then recover just in time to be the generation host — mounting all shards locally while still excluded from the rebalance, which union-only verification accepts before deleting the originals. Abort the encode when a selected source remains skipped after the re-sweep. * fix(ec): batch delete returns retriable 503 when a volume became EC mid-batch If a volume is not EC at the batch-delete classification but is encoded to EC and its .dat deleted before the regular-volume mutation, the mutation returns an exact "not found" that the filer chunk-GC treats as completed, dropping the delete. Recheck EC presence under the mutation lock and return a retriable 503 with the "try again" token so the filer requeues it onto the EC path. * fix(ec): recheck EC state before the regular batch-delete mutation ec.encode mounts EC shards (copied from the .dat) before deleting the originals, so a volume can be EC while its .dat still exists. The batch delete only rechecked EC after a NotFound, so a successful regular-volume delete in that window wrote a tombstone to the soon-removed .dat — the delete was lost and the needle resurrected from the pre-tombstone shards. Recheck has_ec_volume under the write lock before delete_volume_needle and return a retriable 503 so the filer requeues onto the EC path. * fix(volume): make the metrics push test independent of test order test_push_metrics_once asserted the pushed body contains the request-counter family without ever touching the counter — a CounterVec with no children emits nothing, so the assertion only held when another test had already created a labelset in the shared registry. Create one in the test itself. |
||
|
Bruce Zou
|
1dd292fb84 | batch drain delta heartbeat messages (#9914) | ||
|
Chris Lu
|
594fc667d5 |
Cut per-subscriber replay decode and widen replay concurrency (#9917)
* Filter metadata events before unmarshaling them per subscriber Every subscriber unmarshaled every log entry into a full event just to run the path filter, and entries carry complete chunk lists, so a fleet of path-filtered subscribers spends almost all replay CPU materializing events it then discards. A shallow wire scan now extracts just the directory, entry names and rename destination into a skeleton event, feeds the same matcher, and skips the decode for entries the subscriber cannot match. Any scan surprise (malformed bytes, merged duplicate message fields) falls back to the full decode, and the unsynced-events heartbeat keeps firing for skipped entries. * Raise the legacy replay cap The cap was sized when every replay pinned a private chunk reader per source filer. Replays now share decoded chunks, so sixteen needlessly serializes subscriber catch-up; the expensive part stays bounded by the cache's load gate. * Weight concurrent log-chunk loads by size The flat eight-load gate let eight tiny chunks through as reluctantly as eight full ones. Charge each load's chunk size against a 128MB in-flight budget instead: small chunks decode wide open while full-size ones still serialize enough to cap the transient peak. Oversized weights clamp to the budget so they can always acquire. * Propagate heartbeat send failures and reset the skip counter A failed heartbeat send means the stream is gone, so end the replay instead of scanning on. A delivered event also resets the skip counter, keeping the heartbeat cadence relative to the last thing the client actually received. * Share the unsynced-events counter across the prefilter and delivery Two independent counters could starve the heartbeat: alternating drops reset each side before either reached its threshold. One shared counter increments on every dropped entry, prefiltered or not, and only an actual delivery resets it, restoring the original cadence exactly. * Tighten comments * Benchmark the subscription match paths For a thousand-chunk event that the subscriber filters out, the shallow scan matches in 10us and 9 allocations against 175us and 4031 allocations for the full decode. |
||
|
Chris Lu
|
048f9ece2d |
Fix filer metadata-replay OOM under mount reconnect storms (#9901)
* fix(filer): propagate multi-filer metadata log read errors A genuine (non not-found) read error in one filer's log stream was logged and skipped, then the merged cursor advanced past the gap, silently dropping that file's events. Abort the whole replay so the subscriber re-reads from the unchanged position; chunk-not-found still skips. * perf(mount): read persisted metadata log chunks directly from volume servers Set LogFileReaderFn so the filer returns log file references and the mount reads the chunk data itself, instead of the filer reading, decoding, and streaming every persisted entry. Keeps a reconnect storm of many mounts from concentrating hundreds of concurrent log replays in filer memory. * perf(filer): pre-size chunk stream reader buffer to view size The chunk size is known up front, so grow the buffer once instead of letting bytes.Buffer double as the streamed pieces arrive (which transiently overshoots to ~2x per reader). * fix(filer): bound concurrent persisted-log replays Each server-side replay holds an open chunk reader per source filer plus a readahead buffer, so a reconnect storm of clients that predate the metadata-chunks offload multiplies into many GB. Gate replays with a semaphore; abort the acquire when the subscriber's stream is gone so cancelled clients do not pile up parked goroutines. |
||
|
Chris Lu
|
8cc10460b4 |
fix(remote): correct content and permissions when syncing/caching remote objects (#9879)
* fix(remote): reject short reads when caching remote objects A short read from the remote (stale listing size, truncated or flaky response) was silently zero-padded: the S3 and Azure clients pre-size the buffer and discard the downloaded byte count, and the chunk is recorded with the requested size. The cached file then matched the expected size but its tail was NULL, and the entry was marked cached so it never re-fetched. Check the byte count against the requested size in both clients, and add a backend-agnostic guard in FetchAndWriteNeedle. The cache now fails loudly and the entry stays remote-only for a later retry. * fix(remote): match S3 default modes when syncing remote metadata Remote object listings carry no POSIX mode, so synced entries were created with a hardcoded 0644. Against a SeaweedFS remote, whose S3 layer writes objects as 0660 and auto-creates directories as 0771 (0660|0111), the mounted copy ended up 0644/0755 and the permissions visibly diverged from the source. Default to the S3 modes instead (files 0660, directories 0771). The filer derives parent-dir modes from the child as fileMode|0111, so fixing the file default also brings the directories into line. Directory mtimes still reflect sync time: S3 listings don't enumerate directories, so the remote's directory timestamps aren't available. |
||
|
Chris Lu
|
9ede92a7cc |
filer: replicate RECOMPUTE_LATEST pointer updates to peers (#9840)
applyRecomputeLatest wrote the .versions latest-version pointer and the demoted prior version's stamp through UpdateEntry without a following NotifyUpdateEvent, so neither change entered the metadata log. Across filers the pointer then lived only on whichever filer ran the mutation, and ListObjects served by any other filer dropped those objects from a versioned bucket. Emit the events the way PATCH_EXTENDED already does, keeping a pre-update image for the notification diff. |