mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-16 12:16:36 +00:00
0dd33bd7ec012096dffcdfe66b2c5eb2cb069edb
1141
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
08f0ba5564 |
topology: clamp the deleted-vs-total subtractions in volume stats (#10633)
VolumeLocationList.Stats subtracts the deleted figures from the totals to report live size and needle count. Both deleted figures are maintained as counters independent of the totals they come off, so either can transiently exceed its total, and neither subtraction was clamped. Unclamped, the size wraps to ~16 EB. The count is signed so it merely goes negative, but VolumeLayout.Stats converts it with uint64(fileCount), which turns it into ~1.8e19 just the same. Either one swamps the cluster totals behind /dir/status, /vol/status and Topology.CollectionVolumeStats. commandFsMergeVolumes.getVolumeSize had the same unclamped subtraction, where a wrapped size reads as a volume far too large to join any merge plan. Clamped to zero, matching the guards already in CollectionInfo.LogicalSize and the admin server's logical-size accumulator. |
||
|
Chris LuandGitHub
|
44e546a933 |
shell: pick tier.move replica targets with the shared placement picker (#10582)
The command chose destinations by walking its location list in order, so it neither preferred a node near the source nor spread a burst of copies. Replica placement and "this node already holds the volume" move into the Accept predicate; the ranking and the per-pick reservation come from placement. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris LuandGitHub
|
aceab0802e |
placement: move the target picker out of the shell package (#10580)
weed/shell is the CLI command layer: every file there registers a command in init(). Importing it for placement drags that whole surface, and its registration side effects, into callers that run no CLI. The picker now depends only on master_pb and storage/types, with a node type of its own -- smaller than the balancer's Node, which also carries the volumes it holds, and placement never needs those. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris LuandGitHub
|
4b12af036c |
shell: add a reusable target picker for volume moves (#10579)
* shell: add a reusable target picker for volume moves Picks the emptiest node near the source: locality first (same rack, then same data center), emptiest within a tier. Free bytes decide where the cluster reports them, free slots break the tie. The pick is spent in the passed topology, so planning several moves from one snapshot spreads them instead of stacking every one on whichever node started emptiest. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu * shell: order targets on one metric, and reserve the real volume size Comparing some pairs on free bytes and others on free slots is intransitive, so the winner depended on sort order. One node too old to report filesystem bytes now puts every candidate on slots. Reserving the tier average let a batch of large volumes overcommit a destination; callers pass what the move actually consumes. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris LuandGitHub
|
312cfe5ae1 |
Fix volume.merge corrupting every needle it copies (#10565)
* Give volume.merge the needle size the target actually indexes by needleBlobFromNeedle returned the size Append reports, which is Size(n.DataSize) - payload bytes only. The .dat header, the needle map and WriteNeedleBlobRequest.Size all use n.Size, which additionally covers the flags, name, mime and lastModified fields. Every needle volume.merge copied therefore landed with a too-small size. The target indexed it at that length, so every later read failed the header check in ReadBytes with a size mismatch, and on v3 the fresh AppendAtNs stamp landed NeedleHeaderSize+DataSize+NeedleChecksumSize into the blob - exactly on the flags byte - overwriting flags, name size, mime size and the first mime bytes with the top of a timestamp. Needles came back with flags 0x18, no name, no mime and a phantom TTL parsed from two arbitrary timestamp bytes; the ones that decoded as expired 404 and vacuum would drop them. Since merge rebuilds every replica from the merged copy, no clean replica survives. Return n.Size, which Append fills in as it serializes, matching what the normal write path stores via nm.Put. * Reject needle blobs whose size disagrees with their own header WriteNeedleBlob trusts the caller's size for two destructive things: it is what goes into the needle map, and it is where the v3 AppendAtNs stamp is written inside the caller's buffer. A caller passing the payload-only DataSize convention corrupts both, and nothing surfaces until the needle is read back - by which point every replica may already have been rebuilt from it. Parse the blob's own header and refuse the write when the two disagree. Mirrored in the Rust volume server. |
||
|
Chris LuandGitHub
|
b452a5e41b |
s3: honor a bucket owner recorded as an identity (#10567)
* s3: resolve a bucket owner recorded as an identity The admin UI and weed shell record a bucket's owner as an identity name in s3-identity-id and never write the account id the S3 API stores alongside it, so such a bucket looked unowned: its ACL owner fell back to the default admin account, and under the default BucketOwnerEnforced ownership every object uploaded to it was stamped with that account instead of the bucket owner. Resolve the identity to its account when no account id is recorded, in the one place both the bucket metadata and the bucket config derive the owner from. * s3: drop the recorded account when the bucket owner is reassigned Changing the owner of a bucket created through the S3 API left its old account id behind, and that outranks the identity when the owner is resolved, so the new owner never took effect for object ownership or the bucket ACL. |
||
|
Chris LuandGitHub
|
3514925581 |
filer: let a nested path rule turn worm off (#10503)
* filer: let a nested path rule turn worm off mergePathConf ORs the booleans, so worm set on a bucket could never be lifted on a directory under it, while every string field is overridden by the more specific rule. Make worm tri-state instead: unset inherits, set wins. readOnly, fsync and disableChunkDeletion keep the OR, so a nested rule still cannot escape a lock the bucket set. Configurations written before this carry an explicit "worm": false on every rule, because they are marshalled with EmitUnpopulated. Reading those back as an override would quietly drop worm from nested paths, so filer.conf is now stamped with a version and the flag is dropped to unset when the version predates it. * filer: copy the worm value out of the matched rule mergePathConf aliased the pointer into the merged result, so a caller that wrote through it would reach into the stored rule. |
||
|
Chris LuandGitHub
|
c2183566b6 |
ec.encode: name the shard ids an aborted deletion found (#10486)
Counting by node lost the per-node ShardsInfo, and with it the shard ids the old summary printed -- the message an operator gets when the pre-delete check refuses now says only how many shards each node holds. That is the wrong half. A set holding shards 0-9 and one holding 4-13 are both "10 shards", and only the ids say whether what survived can rebuild the volume, or which node to go looking at. Keep the count and list the ids beside it. |
||
|
Chris LuandGitHub
|
c4798979d8 |
ec.encode: count shards wherever they landed before deleting the source (#10483)
generateEcShards writes shards beside the source volume, so encoding a volume that lives on a non-default medium puts them on that medium while -diskType still says hdd. The pre-delete check counted only the -diskType bucket, so it saw a complete set as zero shards, called it unrecoverable and aborted -- leaving the volume as both a .dat and a full shard set, which every later reader then disagrees about. Count by node across disks, as waitForEcShardsToRegister in the same file already does. The spread check is unaffected: it locates shards through collectEcShardBitsByNode and only uses diskType to find free slots. |
||
|
Chris LuandGitHub
|
5536d88fbb |
azure: let the blob endpoint be configured (#10460)
* azure: let the blob endpoint be configured The service url was always derived as <account>.blob.core.windows.net, which leaves out Azure Government, Azure China, and private endpoints. Name the blob service url instead and those accounts become reachable. The url has to be https, since the account key or the bearer token would otherwise travel in the clear. * azure: reject an endpoint that carries no hostname A url like https://:443/ has a host of ":443", so the emptiness check on Host let it through and the request only failed once it reached Azure. The hostname is what has to be there. |
||
|
Chris LuandGitHub
|
3ae4e9c563 |
azure: authenticate with Entra ID instead of a storage account key (#10456)
* azure: authenticate the blob sink with Entra ID Shared account keys have to be distributed and rotated everywhere a sink runs. Leaving account_key empty now falls back to the identity chain, so a workload identity or managed identity carries the authorization instead. * azure: authenticate remote storage with Entra ID The remote storage client demanded an account key and refused to start without one. Fall back to the identity chain when it is absent, and let azure.client_id pin a user-assigned identity. * azure: reject a malformed storage account name The account name is interpolated into the service URL, so a name carrying a "/", "?" or "@" moves the authority elsewhere and an authenticated request follows it. Hold callers to Azure's own naming rule instead. * azure: keep a leftover environment key off the identity path A configured client id asks for Entra ID, but AZURE_STORAGE_ACCESS_KEY still filled in the account key behind it. An old mounted secret would go on authenticating until it rotated, and the failure then blamed the key. * azure: say what the identity path reads from the environment A pinned client id alone is not enough for workload identity: the tenant and the projected token come from the environment, and missing them only surfaces later, when a token is first requested. |
||
|
Chris LuandGitHub
|
6e6255b58e |
shell: accept a context in the volume move helpers (#10415)
LiveMoveVolume and the copy, tail, delete, mark, replicate, and configure helpers around it issued every RPC on context.Background(), so a caller had no way to bound or abort a move once it started. They now take a context, which the exported LiveMoveVolume in particular needs: callers outside the shell drive long moves and want to stop them. The deferred restore in copyVolume runs on a detached, bounded context rather than the caller's. Marking the source writable again is cleanup, and cancelling the copy must not skip it and leave the volume readonly — the same guard balance_task.go already applies for the same reason. Shell commands pass context.Background(): their Do signature carries no context, and changing it would touch every command in the package. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris LuandGitHub
|
47b491b53c |
mount: version open file handles by filer log position (#10403)
* filer: stamp a log position on lookup and remote-cache responses Metadata events are logged after their store write and stamped with the filer clock. Reading that clock before serving an entry therefore gives a timestamp with a causal guarantee: every event at or below it is reflected in the returned entry. Clients caching filer state can use it as the entry's version to order the response against subscription events, including events committed before the call but delivered after it. * mount: version open file handles by filer log position A subscription event refreshing an open handle did a second lookup; a transient failure left the handle pinned to its old entry with no retry, since the subscription cursor had already advanced. The deeper problem is ordering: the handle is a cache written by three unordered channels — the async invalidation worker, local mutation acks, and open-time lookups — and overwriting cached state safely requires knowing which write is newer. The filer log timestamp is that order, and it now travels with every value instead of being derived out of band. Events carry it natively; lookup and remote-cache responses carry the log position stamped before the serving read; mutation acks carry it in their returned event; and the local store pairs each read with a version cursor advanced under the same lock as the store write. Each handle records the version its entry reflects, and one rule replaces the per-site reasoning: state at or below the handle's version is old news and must not be installed. The invalidation itself applies the event's own entry — no lookup, so no transient-failure window — except under a cached parent, where the store entry is the ordered merge of the event and anything applied since, and its version outranks the event's. An uncached parent receives no store writes, so a hit there would be a stale leftover masking the event. A vacated path (delete, rename away) keeps the last entry so unlinked-but-open reads still work. Directory builds version the completed directory at the listing snapshot and re-invalidate buffered events at that version, since their mid-build refresh ran against an incomplete store. The tests replay every race this replaces machinery for: rollback of a newer local flush (queued, cached, and read-through), stale leftovers under uncached parents, the build window including abort, handles opened after an event was queued, events landing mid-lookup, and undelivered events at remote-cache time across a filer failover. * filer: serialize the log position fence with mutations, stamp mutation acks The fence stamped before an unlocked entry read could precede state the read returned: a mutation writes storage first and assigns its event timestamp only at notify time, so a lookup racing that window handed the mount an entry newer than its fence, and the event's later delivery looked like fresh news — destroying dirty pages for a change the handle already had. The mutation handlers already hold an exclusive per-path lock across read, write, and notify; the lookup and remote-cache reads now take it shared around the stamp and the read, making the fence exact: everything at or below it is in the entry, nothing above it is. A no-change update returns success without an event, leaving the mount nothing to fence with even though the response confirms current state. Create and update acks now carry a log position stamped under the same lock, and the mount falls back to it whenever the ack has no event. Also regenerate the VT marshalers, which the earlier generation missed: without them a VT round-trip silently zeroed every log position. * java: sync filer.proto * mount: scope store versions to what they vouch for; atomic handle install The store's version cursor claimed too much. Advanced by local mutation acks and directory listing snapshots, it inflated the version of store reads for unrelated paths whose events the subscription still owed, and those events were then fenced out permanently. The cursor now tracks subscription progress only — events arrive in log order, so everything at or below it has been delivered for every path — and a completed listing records its snapshot as a per-directory floor instead of a global claim. Local acks never touch it: they version their own handle directly. Buffered build events advance the cursor at delivery, since their store write may never happen (abort) while their invalidation is already queued; their read-through directory pairs no store read with it, and rename fragments are applied first. Concurrent first opens raced: a slower opener's older lookup could overwrite the newer entry a faster opener had installed, while the monotonic version kept the newer timestamp — an old entry fenced at a new version, immune to every correcting event. Entry and version are now installed as one decision under the handle map lock, and an install that does not outrank the handle's version is dropped. The remote-cache commit also escaped the fence: it wrote storage and notified without the path lock, so a lookup's shared-locked fence and read could land between the two and hand out the cached state under-versioned. The commit now re-reads and writes under the exclusive path lock, and backs off entirely when the entry changed during the download — the concurrent writer supersedes the cached content. * mount: floors gate store applies; installs respect handle users; renames join the fence A directory floor certifies the listing state as of its snapshot, but a delayed event at or below the floor was still applied to the store — rolling the content back to pre-snapshot state while the floor kept claiming the snapshot version, so the correcting events were fenced out of every future read. Events are now gated against the affected directory's floor, each half of a rename independently. Fences are lower bounds: a listing or lookup can include a mutation whose event has not been delivered yet, and that event later passes every gate carrying state the handle already holds. Such a re-delivery now advances the version without destroying dirty pages or reinstalling the entry — invalidating local writes over a no-op was the real damage in every remaining under-fence window, including the unlocked listing snapshot, which no per-path lock can serialize. The concurrent-open install moved from the map lock to the handle lock every reader, writer, and invalidation synchronizes on, and rejects what cannot improve the handle: dirty state (local writes would be lost), unversioned lookup responses (they cannot outrank anything, and two zero-version opens must not overwrite each other), and anything not strictly newer. New handles are still fully initialized before the map exposes them. Renames committed metadata and emitted events with no path lock, so a lookup could read the renamed state under a fence preceding its events. Both rename handlers now hold the source and destination locks, ordered by path, across commit and notification; descendants of a renamed directory are not individually locked and rely on the no-op re-delivery handling above. * mount: per-entry store versions replace the cursor and directory floors The store's aggregate versions — a global subscription cursor and per-directory listing floors — were versions at coarser granularity than the values they described, and every over-claiming bug in this series traced to that gap: an aggregate vouching for state its source never saw. Each store entry now carries the filer log position of the write that produced it — the event that applied it, or the listing snapshot that inserted it, recorded in the store's key-value space under the same lock as the entry write. The store becomes what the handle already is: a last-writer-wins register with one rule, install only what outranks the current claim. The cursor, the floors, their advancement rules, the pairing ordering constraint, and the floor gating all collapse into that rule. Applies are gated per entry, each half of a rename independently; an unversioned local write clears the claim its content no longer proves; version records lingering after a bulk folder wipe cannot fence a recreate, since a claim only blocks while its entry exists. Listing inserts are stamped at build completion, before the buffered replay so newer replayed events override the stamp. Filer side, the fence dance every versioned read must perform is now a single choke point, fencedFindEntry, so a future read RPC gets the lock-serialized stamp by construction rather than by convention. * mount: judge no-op re-deliveries against an immutable base, not the live entry The equal-state skip compared the incoming event to the live handle entry, but local writes mutate the live entry — size, timestamps, chunks — so a delayed event re-delivering the base the handle was opened with no longer matched, and the installer destroyed the dirty pages and rolled the entry back over nothing new. The handle now keeps an immutable snapshot of the filer state it last installed or acknowledged, refreshed at every install and mutation ack (flush acks snapshot the request entry before the id mapping mutates it), and the no-op judgment runs against that base: an event carrying the base brings nothing, whatever the live entry has diverged to since. * mount: tombstones for versioned deletes, absence floors, copy enrollment Four gaps in the per-entry version protocol, all the same shape: a versioned fact with nothing carrying its version. A deletion is a fact about a path with no entry left to hold it — clearing the record let a delayed older event resurrect the deleted path, permanently, since the deletion's own redelivery is dedup-suppressed. Versioned deletes now leave a tombstone record that fences without an entry; renames tombstone their source the same way. Plain records still only block while their entry exists, so records lingering after a bulk folder wipe cannot fence a recreate. A completed listing proves absences as well as presences: a name it omitted was deleted as of the snapshot, and a delayed create below the snapshot re-creates it. The snapshot is kept per directory strictly as an absence fence, consulted only when a path has neither an entry nor a version record — present entries carry their own versions and never touch it, which is what separates this from the over-claiming floor it replaces. A rebuild against a pre-upgrade filer returns no snapshot; stamping now clears the children's records in that case, so a reinserted entry cannot reactivate the stale claim its previous incarnation left behind and reject valid events below it. Server-side copies installed the copied entry without enrolling in the base protocol, so the copy's own event differed from the stale pre-copy base and destroyed writes made to the destination after the copy. The install now refreshes the base and takes its version from the fenced readback. * mount: deletion facts outlive the cache's knowledge of the entry A versioned delete of a path the store held no entry for recorded nothing, so a delayed older event recreated the path — permanently, with the deletion's redelivery dedup-suppressed. The tombstone is now written whenever a versioned event vacates a path: the deletion is a fact about the path, not about what this cache happened to hold. For an absent entry, the listing's absence floor now speaks whatever older record remains: a tombstone at one position does not exhaust what is known about the path when a newer snapshot has confirmed the name still absent, and an event between the two was slipping past both. A committed copy whose readback failed installed a synthesized base with local timestamps; the copy's real event legitimately differs from it, and was read as foreign state — destroying writes made to the destination after the copy. The handle now marks that its own event is en route and adopts that event's state as the base without touching the live entry or the dirty pages; the adoption is one-shot, so a genuinely foreign event still invalidates. * mount: authoritative acks cancel pending event adoption; tombstones scoped and pruned The copy-event adoption flag could outlive its purpose: a flush after the failed readback installs a newer base and advances the version, the copy's own event is then version gated without consuming the flag, and the next genuinely foreign event was silently adopted — base advanced, live entry and dirty pages untouched — leaving the mount to later overwrite that remote change. Every local acknowledgment now installs its base through one helper that also cancels any pending adoption: the ack supersedes the mutation the adoption was waiting for. Tombstones were written for every versioned delete under the mount and survived directory eviction by design, growing LevelDB with historical deletions on delete-heavy mounts. They are now scoped to directories whose cached state the fence actually protects — an uncached parent never serves from the store nor applies the resurrecting insert — and a completed listing prunes the direct-child tombstones its absence floor supersedes, leaving only those above the snapshot. The store gains a key-prefix visitor for the sweep. * mount: acked saves install their value; trailer snapshots; direct-child prune range A version must never advance without its value. saveEntry stamped any open handle with the acknowledgment's version, but a handle opened while the save was in flight holds the pre-mutation entry — stamping it fenced out the events carrying the state it lacked, permanently, with the local apply performing no invalidation and the redelivery deduplicated. The acknowledged entry is now installed together with its version, through the same guarded install the racing-open path uses: under the handle lock, only when it outranks the handle, never over dirty local writes. Empty listings return no in-band snapshot — a snapshot-only response would be read as an entry by older consumers — so directories that end empty gained no absence floor and their tombstones were never pruned. The filer now sends the snapshot in the stream trailer, which older clients ignore, and the client reads it when no in-band snapshot arrived. Empty directories get real floors, their tombstones prune, and their buffered replays gain the snapshot filter instead of the replay-all fallback. Version records now encode the parent directory and name separated by a NUL, making a directory's direct children one contiguous key range: the tombstone prune scans exactly them under the cache lock, instead of walking every descendant record — the whole store, for root. * mount: fix dirty-page loss, uid/gid base, download race, copy adopt, leak; dedup Correctness fixes from the versioned-invalidation review: - A foreign delete/rename-away of a file held open with unflushed local writes destroyed the dirty pages unconditionally. A process may keep writing to an unlinked-but-open file and those writes were already acknowledged; preserve the pages when the handle is dirty. - downloadRemoteEntry stored the handle's base with filer-side uid/gid while every candidate it is later compared against is in local form, so under a non-identity UidGidMapper an unchanged re-delivery looked foreign and force-destroyed dirty pages. Map the base to local. - downloadRemoteEntry wrote the entry/base/version triple under only the handle's shared lock, so two concurrent reads of the same remote-only file could tear it. Serialize the install with a dedicated mutex (invalidation is already excluded by the exclusive handle lock). - A committed server-side copy whose readback failed adopted the FIRST event past the version gate as its base; a foreign write delivered first was silently swallowed. Adopt only an event whose content matches the synthesized base — the copy's own event — and install any other normally. - The deferred-create path relied on AcquireFileHandle installing the passed entry on a pre-existing handle, which the version rework dropped. Restore that install in the compat wrapper; the versioned open path keeps its gated install. Growth and hot-path cost: - Per-entry version records and tombstones leaked when a directory was evicted or read-through without a rebuild. An uncached directory gates its own inserts, so its records fence nothing; clear a directory's child version records when it is wiped for eviction. - FindEntry paid for the version KvGet on every lookup/getattr cache hit and threw it away. FindEntry now reads only the entry; the hot lookupEntry cache-hit path skips the version entirely. Cleanups: - Extract ackVersionTsNs over the shared response interface, replacing the metadata-event-else-log-ts snippet copy-pasted at four ack sites. - Extract acquireRenamePathLocks, replacing the verbatim sorted two-path lock fence in both rename handlers. * mount: no resurrection on foreign delete, version no-event acks, gate downloads, tighten copy adopt Follow-ups to the review patches: - Preserving dirty pages on a foreign delete let the next flush pass the isDeleted guard and CreateEntry, resurrecting the remotely-unlinked name. Mark the handle deleted in the vacate branch: the open fd can still read its buffered writes, but a flush no longer recreates the file. - A no-event acknowledgment (log fence only) synthesized a metadata event with TsNs 0, so the cache stored the entry unversioned and an older subscriber event rolled it back. Stamp the synthesized event with the ack's log position at all four ack sites. - downloadRemoteEntry serialized its install but did not check the version, so an older response arriving last overwrote the entry/base while the monotonic version kept the newer value, fencing corrections out. Install only when the response is at least as new as the handle. - sameEntryContent compared only size and chunks, so a foreign chmod with unchanged content was adopted as the copy's own event. Compare everything except server-assigned timestamps, so a metadata-only foreign change installs instead. * mount: trim comments to the non-obvious why The versioning work accumulated multi-line comment blocks restating what the code says. Keep the constraint a reader cannot derive — why a fence is exact, why a version must not advance without its value, why an uncached parent's records fence nothing — and drop the rest. * mount: distinguish rename from delete, tighten the download and adopt gates - A rename emits a nil old-path invalidation just like an unlink, so the vacate branch marked the handle deleted and later writes through the already-open descriptor were skipped instead of persisted. Carry the delete/rename distinction on the invalidation and mark only an actual delete. - The remote-download install accepted an unversioned response regardless of the handle's version, so during a rolling upgrade a delayed response could install stale content under a newer version. Require the response to be at least as new, with one exception: a handle still lacking local chunks takes the content anyway — it cannot read without it — but does not claim the response's log position. - Copy-event adoption returned without installing, so a foreign touch arriving before the copy's own event lost its timestamps. Content is unchanged either way, so the dirty pages stay valid; a clean handle now takes the entry, while a dirty one keeps its diverged version. * mount: one directory floor instead of a record per child; agree on TTL Review feedback: - Build completion wrote one KV record per direct child inside the cache write lock, so a large directory stalled every other cache operation for O(children) store writes. The directory's listing snapshot already covers every child it saw; make that floor the version for any child without a record of its own, and a child earns a record only when a later event touches it. One map write per build replaces the per-child writes, with the same fencing. - The presence probe read the store directly and so counted a TTL-expired entry as present, judging the path by a record describing content that has logically vanished. It now applies the same expiry the read path does, and an expired path falls back to its directory floor. - Preserve ErrNotFound identity when the commit-time re-read finds the object deleted, so callers still surface a 404. - Assert the rename-away source fence timestamp in the invalidation test. Also record the tombstone ceiling: distinct deleted names in a cached directory accumulate until it is rebuilt or evicted, which prunes everything at or below the new snapshot. * mount: pin the fence's clock domain instead of letting skew decide A log-position fence is stamped by one filer's clock under that filer's in-process lock, so comparing it to an event another filer logged is comparing two unrelated clocks. The two error directions are not equally costly: applying an event the fence already covered is a re-apply the base-equality check absorbs, while skipping one it does not cover leaves the handle holding exactly the state the event was meant to correct, with the subscription cursor already past it — the unhealable staleness this whole PR exists to remove. So refuse to guess. Fences now carry the signature of the filer that stamped them, and a handle records it alongside the position. An event is only fenced out when the filer that logged it is the one that stamped the fence — the logging filer appends its own signature, so its presence identifies the clock domain. Events from any other filer are applied. Positions taken from events keep comparing as before; the subscription already delivers those in order. The invalidation callback takes a struct now: it carries the path, entry, position, delete/rename distinction, and signatures, and was about to need a fifth positional parameter. * mount: follow a foreign rename; key page invalidation on content, not equality - A rename's old-path invalidation now carries the destination, and the handle follows the file there: an open fd tracks the inode, and leaving it on the old path made its next flush recreate that name instead of updating the renamed file. - Dirty pages overlay content, so only a content change invalidates them. Keying that on exact equality meant any timestamp-only event destroyed them, which the copy-adoption marker existed to paper over — a foreign touch could consume the marker and leave the copy's own event to drop the post-copy writes. Comparing content instead makes the marker unnecessary, so it is gone: a metadata-only event keeps the overlay, and a dirty handle keeps its diverged entry unless foreign content supersedes it. - A remote download response that is merely older is now refused even when the handle still lacks chunks; only an unversioned one is taken (and claims no position), since an older response's content predates what the handle reflects. - A refused or unversioned download no longer publishes to the metadata cache, where a zero-position event would clear the entry's version and let an older subscriber event roll the cache back. * mount: page invalidation keys on content alone; unversioned writes claim no position - sameEntryContent compared everything but timestamps, so a foreign chmod, chown, or xattr change counted as a content change and destroyed the dirty-page overlay. It was strict only to serve the copy-adoption marker, which is gone; its one caller now asks the question it actually needs — did the bytes change — so metadata-only events leave the overlay alone. - A rename over an existing file destroys that file, but its open handle was left live and still pointed at the name the renamed source now occupies, so its flush could overwrite it. MovePath already reports the displaced inode; mark that handle deleted. - An acknowledgment was refused whenever its position was numerically lower, even when a different filer stamped the fence it lost to. Two known, differing signatures mean unrelated clocks, so the comparison no longer applies there; unknown signatures still compare as before. - A local write with no log position behind it now records that explicitly instead of deleting its version record. Absence means the directory listing covers the path, which is why the snapshot floor applies; local content the listing never saw must not inherit it, or the events that would correct it are fenced out. * mount: widen the existing lookup functions instead of forking WithVersion twins The versioning work grew a parallel function for every accessor that needed to return a log position — lookupEntryWithVersion beside lookupEntry, maybeLoadEntryWithVersion beside maybeLoadEntry, FindEntryWithVersion beside FindEntry, AcquireFileHandleWithVersion beside AcquireFileHandle, advanceEntryVersion beside advanceEntryVersionTsNs, plus a getPbEntryWithVersion wrapper and an InsertListedEntriesForTest hook. Two names for one operation is two places to keep in step, and the split let callers pick the one that happened to compile. Each pair is now the single original name carrying the position, with callers that do not want it discarding it. filer_pb.GetEntry returns the fence its response already carried rather than a mount-side wrapper re-issuing the lookup, and InsertEntry takes the position its content reflects rather than a test-only twin that inserted without one. The one behavioural knot the merge exposed: AcquireFileHandle had been installing the entry on a pre-existing handle only in its unversioned form, which conflated 'the caller is authoritative' with 'the lookup had no version'. Deferred create is the only caller that means the former, so it now installs explicitly and the map function just acquires. |
||
 
|
490379bff3 |
Add codespell support with configuration and typo fixes (#10393)
* Add GitHub Actions workflow for codespell on master * Add rudimentary codespell config * Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers like allLocations, publishErr, ReadInside, FlushInterval. Also skip templ-generated *_templ.go files, and whitelist a handful of short/domain-specific words (visibles, fo, te, ser, bject, unparseable, keep-alives, tread, anc, ue) that show up as false positives across the tree. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix ambiguous typos and protect false positives Fixes typos that codespell reports with multiple candidate suggestions (so `codespell -w` cannot auto-apply them), plus one inline pragma and one config entry to protect legitimate identifiers. Manual fixes (single correct answer chosen from context): - pattens -> patterns (5x) in filer/upload/shell flag help strings - finded -> found (2x) in tarantool storage.lua comment - spacify -> specify (2x) in helm chart values.yaml comment - wether -> whether in skiplist.go docstring - simpe -> simple in mq schema test case name False-positive protection: - Add `//codespell:ignore` next to `source GET's` (possessive of HTTP verb) in s3api_object_handlers_copy_stream.go - Whitelist `auther` in .codespellrc — it's a local variable meaning "authenticator" in weed/security/tls.go, not a typo of "author". Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Extend codespell ignore list: .git-meta path and thirdparty groupId Also skip `.git-meta` (scratch dir for commit messages that may contain typo words verbatim) and whitelist `thirdparty` — it appears as the literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms and cannot be renamed. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w Auto-applied fixes to the 44 remaining single-suggestion typos across docs, comments, log messages, tests, config, and one Java pom. === Do not change lines below === { "chain": [], "cmd": "uvx codespell -w", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^ * Revert breaking codespell fixes; whitelist unknwon and atleast Two of the auto-applied `codespell -w` fixes were false positives that would break the build/tests: - go.mod: `github.com/unknwon/goconfig` is a real Go module path — the upstream author's GitHub handle is literally `unknwon`. Renaming to `unknown` would fail dependency resolution. - test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}: `atleast` is a literal CLI mode value (a string constant compared and passed as a positional argument). Rewriting to `at least` splits it into two arguments and breaks the mode check. Reverted those files and whitelisted both words in .codespellrc so future runs won't re-suggest the same broken fixes. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
 
|
ca06589d64 |
volume.fix.replication: add a well-placed replica before deleting a misplaced one (#10364)
* volume.fix.replication: add a well-placed replica before deleting a misplaced one A misplaced volume with no surplus replica (replica count == copy count) used to have its misplaced replica deleted first, and the replacement copied only on the next pass. That drops the volume below its intended durability, and permanently so when no destination can accept the replacement copy. Now such volumes get a well-placed replica first, and the misplaced one is trimmed as surplus on a later pass, following the same fulfill-before-delete principle as volume.tier.move (#8950). The classification switch is extracted into classifyReplicaSet so the add-wins-over-trim priority is unit-testable. Also stop the fix loop when an -apply pass makes no progress (nothing copied, nothing deleted); previously an unplaceable under-replicated volume made the loop spin forever, re-collecting the topology every 15 seconds. And propagate ewg.Wait() errors instead of swallowing them. * volume.fix.replication: drop unused allLocations from deleteOneVolume --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
02df1cf428 |
shell: cluster.ps lists s3 servers (#10359)
* shell: cluster.ps lists s3 servers * explicit returns in listClusterNodes helper |
||
|
Chris LuandGitHub
|
564803becd |
shell: show who holds the cluster lock (#10353)
* regenerate master_grpc.pb.go with protoc-gen-go-grpc v1.6.2 The other generated pb files are already on v1.6.2; this one was stale. * shell: keep unlock from racing the lease renewal A renewal RPC in flight while ReleaseLock runs re-creates the lock on the master after the release deletes it, and can blank the client name if the renewal reads it mid-release. The stale-token release is then ignored, so the lock stays held (sometimes anonymously) until it expires. Serialize the renew and release RPCs, and set the client name before flipping isLocked so the renewal never sends a partial acquisition. * shell: restart lease renewal after a failed renewal The renewal goroutine exits on error but never cleared its running flag, so later locks in the same process were never renewed and silently expired after ten seconds. * shell: show who holds the cluster lock A blocked lock command gave no hint that another client holds the lock (the refusals only surfaced at -v=2), and cluster.status reported the shell's own lock state as if it were the cluster's. Add a GetAdminLockStatus RPC to the master so lock prints the holder before blocking and cluster.status shows the actual cluster-wide holder. Both degrade silently against masters without the RPC. * shell: bound admin lock RPC attempts with timeouts The lease, renew, release, and holder-status calls all ran without a deadline, so an unresponsive master could hang the renewal goroutine, an unlock (which now waits on the renewal mutex), or the shell prompt. Give each attempt its own short context; the retry loops still resolve a fresh leader on the next try. * master: reject admin token release on non-leaders A follower holds no lock state, so it answered a release with success while the leader kept the lock until expiry. Refuse like LeaseAdminToken does so the client can try the leader instead. * shell: leave the lock release call unbounded A release cut short by a deadline leaves the lock held on the master until it expires, so a slow master would turn every unlock into a ten-second ghost lock. Restore the single fire-and-forget attempt; the timeouts stay on the lease and renew paths, where a stalled call forfeits the lease anyway. * shell: release only the token unlock started with A RequestLock racing a slow release (the admin presence lock does this on shutdown) could have its freshly acquired token sent in the release request or zeroed by the trailing stores. Capture the token once under the mutex and compare on clear so a concurrent acquisition survives an in-flight unlock. |
||
|
Chris LuandGitHub
|
25ab4c3cac |
preserve Content-Encoding for remote-mounted objects (#10340)
* remote storage: carry Content-Encoding into mounted entries A RemoteEntry now records the remote object's Content-Encoding, and every path that materializes a local entry from remote metadata (lazy fetch, lazy listing, remote.mount, remote.meta.sync, remote.cache) stamps it into the entry extended attributes, so HTTP and S3 HeadObject/GetObject return the header. GCS and Azure populate it on listing and stat; S3 only exposes it via HeadObject, so listings leave it empty. * remote storage: set Content-Encoding when uploading to the remote An entry carrying Content-Encoding in its extended attributes (a native S3 upload, or a value pulled from the remote) now keeps it when filer.remote.sync or remote.copy.local writes the object to GCS, S3, or Azure, instead of silently dropping it. * gcs: read remote objects without decompressive transcoding GCS transparently decompresses gzip-encoded objects on download, which ignores range requests and returns byte counts that disagree with the tracked RemoteSize. Request the stored bytes instead; chunked reads of gzip-encoded objects then behave like any other object. * remote storage: track Content-Encoding presence so removals propagate A listing that does not report encodings (S3) leaves the field unset and the local header untouched, while an authoritative report of no encoding (GCS, Azure, any stat) now clears a previously stamped header instead of leaving it stale. remote.cache also schedules a metadata update when only the reported encoding changes. * remote storage: propagate Content-Encoding on metadata-only updates filer.remote.sync routes same-content changes through UpdateFileMetadata, which only touched custom metadata (GCS, Azure) or tags (S3), so a Content-Encoding change in the extended attributes never reached the remote object's real header. GCS now patches contentEncoding alongside the metadata, and Azure reissues the blob's HTTP headers with the new value, carrying the others over since the call replaces the full set. S3 stays tags-only: changing the header there means rewriting the object, which the sync already does whenever content changes. * remote.meta.sync: optional per-file stat for listing-omitted metadata S3 listings carry no Content-Encoding, so entries synced from them never learn it and the lazy-stat path never runs once an entry exists. With -statFiles, each new or changed file whose listing left the encoding unreported is stat-ed before reconciling, and the stat-derived value is persisted so the next run only stats files that changed. Off by default: it costs one remote request per file, and GCS and Azure listings already carry the encoding. * s3: apply metadata-only Content-Encoding changes with an in-place copy Content-Encoding is S3 system metadata, so the tags-only metadata update silently left the object's real header untouched. When the encoding differs, reissue the object as a self-copy with replaced metadata, carrying the content type and configured storage class like a fresh write does. CopyObject caps at 5 GiB; beyond that the change is logged and applies on the next content write. * azure: skip the metadata call when user metadata is unchanged An encoding-only change reissues the blob's HTTP headers; sending the unchanged user metadata alongside it wastes a round trip and bumps the blob's ETag once more than needed. * s3: carry existing object metadata through the encoding copy The replace directive drops everything not resent, and a mounted entry usually has no local mime or user metadata, so the in-place copy wiped the object's Content-Type, Cache-Control, user metadata, encryption settings, and storage class. Read them back with a HeadObject first and carry them over, overriding only what SeaweedFS manages: the encoding, a locally set mime, and the configured storage class. S3 reports Expires as a string while the copy input wants a time, so it is parsed and skipped when malformed. |
||
|
Chris LuandGitHub
|
10cdaf3818 |
Introduce weed shell command ec.check.replication. (#10328)
* Introduce weed shell command `ec.check.replication`.
This command performs a quick check of EC volume shard replication, reporting
volumes whose shards are over- or under-replicated. Each volume is checked
against its own data+parity ratio, obtained via
erasure_coding.EcShardsVolume{Data,Parity}Shards, so builds that derive the EC
ratio per volume report custom ratios correctly.
The name follows the shell's convention (cluster.check, volume.check.disk); the
closest normal-volume counterpart is volume.fix.replication.
* shell: ec.check.replication reports mixed under+over-replication in both lists
|
||
|
Chris LuandGitHub
|
013281b498 |
shell: volume.check.disk -resurrectMissingNeedles for never-vacuumed replicas (#10316)
An absent needle is normally indistinguishable from a vacuumed delete, so check.disk skips it and replicas diverged by replication failures cannot be reunited. But a replica with compaction revision 0 has never been vacuumed: every delete it processed still holds its tombstone, so a needle absent there is provably a missing write. The new flag resurrects absent needles in exactly that case. The receiving replica's live compaction revision is read after the index snapshot and must be 0; vacuumed replicas keep the skip, now with the revision in the message. The decision is passed per direction since pass 2 runs pairs concurrently. Resurrected needles count toward -nonRepairThreshold as before. |
||
|
Chris LuandGitHub
|
c1a1e3c1e3 |
shell: volume.tier.upload keeps volume replicas (#10314)
* volume: copying a remote-backed volume only needs space for the index VolumeCopy sized its target-location check by the source .dat even when that .dat lives in a cloud tier and only .idx/.vif land locally, so re-replicating a tiered volume demanded the full remote size in free disk. Require the index size instead. * shell: volume.tier.upload keeps volume replicas Tiering a replicated volume deleted every replica but the upload source, leaving one server holding the only .idx and the only .vif that knows the remote object key — losing that server orphaned the volume even though its data sat intact in the cloud. Replicate the uploaded .idx/.vif onto the other replica servers instead (VolumeCopy skips the .dat for remote-backed volumes), so all replicas serve reads from the same remote object and the volume keeps its replica count. An already-tiered replica is preferred as the upload source, so a rerun after a partial failure reuses the existing remote object instead of uploading a second copy under a new key. * shell: group tier upload locations instead of re-prepending * rust volume: copying a remote-backed volume only needs space for the index Mirror the Go VolumeCopy change: size the free-location check by the source .idx when the .dat lives in a cloud tier, since only .idx/.vif land locally. |
||
|
Chris LuandGitHub
|
e7be6bb2f8 |
filer: allow clearing a bucket read-only flag stuck after quota removal (#10310)
* filer: allow clearing a bucket read-only flag stuck after quota removal Once s3.bucket.quota.enforce marks a bucket read-only, removing or disabling the quota orphans the flag: enforcement skips buckets with a non-positive quota, and fs.configure merges booleans with OR so -readOnly=false could never turn it off. The only way out was deleting the whole path rule. - s3.bucket.quota -op=remove/disable and the admin server quota update now lift the read-only flag on the bucket's path rule - fs.configure now honors explicitly passed false boolean flags (-readOnly, -fsync, -worm) instead of OR-merging them away * filer: ClearBucketReadOnly reports unchanged when the save fails |
||
|
Chris LuandGitHub
|
298ab35fd7 |
shell: default batch size for ec.encode and ec.decode (#10308)
* shell: ec.encode batches 10 volumes by default Encoding a whole collection as one batch means a late failure leaves everything half-converted. Default -batchSize to 10 so each batch is encoded, rebalanced, verified, and its originals deleted before the next starts. -batchSize=0 keeps the all-at-once behavior. Batch progress messages now print only when there is more than one batch, so small runs read as before. * shell: ec.decode decodes in batches, refreshing topology in between ec.decode walked every volume off the single topology snapshot taken at startup, which goes stale as earlier decodes move shards around and create volumes. Decode 10 volumes per batch by default, re-collecting the topology and rebuilding the free-space accounting between batches. -batchSize=0 keeps the single-snapshot behavior. |
||
|
Chris LuandGitHub
|
5c511c4894 |
ec.encode: don't rebalance against a topology that predates the new shards (#10303)
Mounting freshly generated shards notifies the master asynchronously, while the post-encode rebalance plans from a fresh VolumeList snapshot; a snapshot that predates the delta shows zero shards, so the balance plans no moves and silently succeeds, and the original .dat is deleted with all shards on the generation host. Poll until every encoded volume reports a full shard set before balancing — across all disk types, since fresh shards register under the source volume's disk type, and scoped to the newest encode generation so an orphaned older generation neither satisfies the wait nor defeats the clump check. As defense in depth, refuse to delete originals when a volume's shards still sit on one node while another node has free EC slots. |
||
 qzhelloandGitHub
|
95023af489 | fix(shell): print markVolumeWritable/markVolumeReadonly based on the writable flag (#10283) | ||
|
Chris LuandGitHub
|
02b5b66a3f |
topology/balancer: share replica selection between shell and workers (#10276)
Move pickOneReplicaToCopyFrom, pickOneReplicaToDelete, pickOneMisplacedVolume, and isMisplaced into weed/topology/balancer, following satisfyReplicaPlacement. Selection functions take the shared balancer.Replica shape and return indices, so the shell keeps its own replica type; behavior is unchanged and covered by the existing shell tests. |
||
|
Chris LuandGitHub
|
78e428758b |
volume.fix.replication: parallelize under-replicated volume copies (#10275)
* volume.fix.replication: parallelize under-replicated volume copies Fan out fixOneUnderReplicatedVolume up to -maxParallelization at a time. Destination selection and free-slot accounting stay atomic behind a scheduler mutex, so concurrent fixes see each other's reservations; a failed copy returns its reserved slot. A per-server in-flight cap (-maxParallelizationPerServer, default 1) keeps many simultaneous copies from swamping a single destination server; when every eligible destination is at the cap the fix waits for a slot instead of failing. * volume.fix.replication: isolate the scheduler's location ordering Clone allLocations before the per-volume fan-out so the scheduler's re-sorting cannot alias the slice shared with the concurrent delete phases, and make the test's per-iteration location copy explicit. |
||
|
Chris LuandGitHub
|
a9cfbd8d3a |
s3: tear down the emptied .versions directory on last-version delete; drain existing residue (#10278)
* s3: routed last-version delete removes the emptied .versions directory The routed versioned delete (routedDeleteSpecificVersion) repoints the latest pointer and deletes the version file, but unlike the lock-path fallback (updateLatestVersionAfterDeletion) it never tears down the .versions/ directory it just emptied. The residue keeps the key's read path in the self-heal rescan loop: every GET of the deleted key logs event=surfaced plus a GetObject error until the background EmptyFolderCleaner gets to the directory — at least two minutes away on its delay queue, and possibly never (the queue is in-memory, bounded, and gated on the bucket's allow-empty-folders policy). Veeam's lock arbitration probes deleted lock keys continuously, so those windows are always open and the log spam is chronic. ObjectMutation DELETE gains remove_empty_parent: after the child delete, the filer best-effort removes the parent directory in the same locked transaction. Non-recursive on purpose — a concurrent write that lands a new version fails the removal instead of being lost with it. The routed last-version delete sets it on the version-file DELETE, matching the lock-path fallback's contract. Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv * s3: drain empty .versions residue on read heal and in s3.versions.audit Directories already stranded by pre-teardown deletes (or dropped from the EmptyFolderCleaner's bounded in-memory queue) previously re-entered the self-heal rescan on every GET forever: the heal only cleared the pointer and nothing ever removed the directory, and s3.versions.audit counted the state as clean. When the heal rescan finds no remaining version, remove the directory outright (non-recursive, so orphan children still block and fall back to the pointer clear) and log event=healed mode=empty_dir_removed; the next GET takes the clean not-found path. The audit gains an empty category so the residue is visible, and -heal removes such directories in bulk. Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv |
||
|
Chris LuandGitHub
|
11fd20e2b0 |
s3.bucket.list: measure quota usage by logical size (#10274)
Align the usage percentage with s3.bucket.quota.enforce and the admin UI, which measure quota against the live data size (size minus un-vacuumed deleted bytes). Also print the logical size so both the raw and live views stay visible. |
||
|
cfb46ee19f |
volume.balance: rank by physical disk usage (#10271)
* volume.balance: rank by physical disk usage * volume.balance: keep -byDiskUsage ranking on one scale across the fleet A server that does not report disk bytes ranks at whole volume-equivalents while reporting servers stay below 1.0, so during a rolling upgrade the balancer drains nearly empty old-build servers onto physically fuller ones. Decide the scale once: rank by physical used percent only when every server reports disk bytes, otherwise fall back to the data-size ranking for all. Normalizing the fallback by maxVolumeCount instead would reintroduce the over-configured-maxVolumeCount distortion this flag exists to avoid. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
 
|
e64d01825f |
feat(shell): add -delete option to remote.copy.local (#10228)
* shell: add -delete option to remote.copy.local Add a -delete flag to remote.copy.local that removes files and directories from remote storage when they no longer exist locally, similar to rsync --delete. This makes the command usable for scheduled one-shot backups that also propagate local deletions. - -include/-exclude patterns also limit which remote files are deleted - size/age filters only apply to copying, since remote entries have no local attributes to filter on - orphaned remote directories are removed after their contents, deepest first, and only when no name filter is set (a recursive RemoveDirectory could otherwise remove intentionally kept files) - deletion is skipped entirely if any copy failed - -dryRun shows DELETE lines for review before committing to anything Fixes seaweedfs/seaweedfs#8609 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * shell: fix remote.copy.local -delete deleting files outside -dir Traverse lists remote objects by key prefix with no delimiter, so a -dir pointing at a subdirectory also matches siblings that merely share its name prefix (foo -> foobar). Those are not under the local traversal root, so -delete treated them as extraneous and removed them. Scope deletion candidates to paths under dirToCopy. Also drop the directory-removal path: RemoveDirectory is a no-op on every backend and Traverse never emits directory entries, so it only ever printed success for work it never did. --------- Co-authored-by: Jason Lin <jason@jtx.com.tw> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
292c7493fa |
s3: enforce bucket quota on logical size and surface read-only state in Admin UI (#10224)
* s3: enforce bucket quota on logical size, not un-vacuumed physical size A bucket full of deleted/overwritten objects awaiting vacuum went read-only while its live data stayed under quota, because enforcement used the raw single-copy volume size with garbage included. Subtract DeletedByteCount via a LogicalSize() helper in the auto-enforce loop, the s3.bucket.quota.enforce command, and the bucket_size_bytes metric (labeled logical but counting garbage too). Deleting objects now relieves quota immediately and enforcement matches the UI usage figure. * admin: surface bucket read-only state in the S3 buckets UI Read the read-only flag quota enforcement writes to filer.conf and show it as a badge in the bucket list and a Status row in the details modal, so an operator can see why writes are being rejected. |
||
|
qzhelloandGitHub
|
2c980fb468 | fix(shell): pass collection in ec.shard.unmount --delete request (#10219) | ||
|
Chris LuandGitHub
|
17af32f3ff |
s3: paginate ListBuckets and serve it from a bucket owner index (#10214)
* s3: paginate ListBuckets with max-buckets, continuation-token, and prefix ListBuckets buffered every bucket entry into one slice and one XML body, which falls over with very large bucket counts. Page through the filer listing instead, cap each response at 10000 buckets like AWS, and honor max-buckets, prefix, and an opaque keyset continuation-token. * s3: maintain a bucket owner index under /buckets/.system/owners Map each bucket owner to its buckets as zero-length entries at /buckets/.system/owners/<owner>/<bucket>, with Crtime mirroring the bucket's creation time. The bucket handlers write the index synchronously, the /buckets metadata subscription reconciles changes made elsewhere (weed shell, other gateways, direct filer operations), and a startup backfill indexes pre-existing buckets before writing a ready marker. Owner names are path-escaped so no identity name can escape the index directory. * s3: serve ListBuckets from the bucket owner index Once the owner index is ready, non-admin identities list their owned buckets straight from it, merged with any buckets their legacy actions name explicitly, so ListBuckets costs O(own buckets) instead of a scan of the global /buckets directory. Admins, identities with a bare List grant or wildcard action patterns, and policy-authorized identities whose grants cannot be enumerated keep the paged scan; policy-routed identities get their owned buckets, matching AWS ListBuckets returning only the caller's buckets. * s3: keep dot-prefixed names under /buckets out of bucket surfaces Dot-prefixed entries (.system) can never be valid bucket names, so refuse to resolve them as buckets and skip them in the shell bucket listing, matching what ListBuckets and the admin UI already do. * test: cover ListBuckets pagination and the owner index end to end * s3: fail closed on a nil identity when routing ListBuckets * s3: decide the IAM authorization mechanism in one place VerifyActionPermission and the ListBuckets owner-index routing each re-derived the session-token / attached-policy / legacy-actions split; extract the decision so the two cannot drift. * s3: heal the owner index on concurrent bucket recreation too The mkdir-lost-the-race path answers BucketAlreadyOwnedByYou just like the up-front existence check, so give it the same index repair. * s3: drop owner-index records for buckets deleted during backfill A bucket removed between the backfill reading its page and writing the index record became a permanent phantom in its owner's listing: the delete's own cleanup ran before the record existed. After indexing each page, re-list the same name range and remove records whose bucket is gone; deletes landing after the re-list find the record and remove it themselves. * s3: add ContinuationToken and Prefix to the ListBuckets schema Keep AmazonS3.xsd aligned with the generated ListAllMyBucketsResult so a regeneration does not drop the pagination fields. |
||
|
Chris LuandGitHub
|
01406e661a |
weed shell: add ec.shard.unmount command (#10204)
Unmount, and optionally delete, EC volume shards from the shell, so broken or over-replicated shards can be handled without stopping volume servers and deleting files by hand. Default action is unmount; --delete also removes the shard files. Targets resolve against the live topology, with shardId:host:port to disambiguate co-located or over-replicated shards. Dry-run by default; pass --apply. |
||
|
Chris LuandGitHub
|
746ed82662 |
remote.meta.sync: sync directories and remove files deleted from remote (#10184)
* remote.meta.sync: materialize directory entries, including empty ones Pull metadata by walking the remote tree one directory level at a time with a delimiter, so subdirectories, including empty ones, are listed as their own entries and created locally. The previous flat listing only returned files, so empty remote directories never appeared locally and non-empty ones only existed as filer-synthesized parents. * remote.meta.sync: remove local metadata for entries deleted from remote After reconciling each directory, drop local entries whose remote source is gone: files are deleted outright, and a directory removed from the remote is descended into so its remote-backed children are cleaned while local-only entries are kept. remote.meta.sync exposes -delete (default on) and remote.mount.buckets reconciles the same way; a plain remote.mount stays additive. * remote.meta.sync: reconcile type swaps and prune emptied directories - when the remote swaps an entry's type (file <-> directory), drop the stale local entry and recreate it with the right type; local-only entries are left alone - mark synced directories remote-backed and clean a directory removed from the remote locally, deleting it once it holds no local-only entries, instead of re-listing the missing remote path - treat a differing remote size or mtime, not only a newer mtime, as a change worth pulling |
||
|
Chris LuandGitHub
|
05b4b5bf56 |
ec: expose force_deleted_needles_check in ScrubEcVolume RPC and shell (#10176)
* ec: expose force_deleted_needles_check in ScrubEcVolume RPC and shell FULL EC scrubs can opt into strict deleted-needle verification via the -forceDeletedNeedlesCheck shell flag, off by default since it can report false positives when EC indexes disagree. Rejected for non-FULL modes. The Rust volume server parses the new field and ignores it: its FULL scrub verifies shards via RS parity, not per-needle reads. * volume: require admin auth for ScrubEcVolume ScrubEcVolume ran unauthenticated while its sibling ScrubVolume, and the rest of the mutating volume handlers, gate on checkGrpcAdminAuth. Close the gap so an EC scrub can't be triggered anonymously. * shell: reject ec.scrub -forceDeletedNeedlesCheck outside full mode Fail in the client before fanning out to every volume server, instead of erroring halfway through once the servers reject the request. |
||
|
Chris LuandGitHub
|
b872d5e683 |
balance: extract the bytes-aware density metric to a shared package (#10174)
* balance: extract the bytes-aware density metric to weed/topology/balancer The shell's volume.balance ranks servers by a bytes-aware density (used volume equivalents over free capacity). Move that math into the shared balancer package (VolumeDensity / DensityRatio / DensityNextRatio) so the maintenance worker can adopt the same metric next. Shell behavior is unchanged. * balance: rank a server with no free capacity as the fullest DensityRatio/DensityNextRatio divided by capacity, so a server past its slot limit (negative capacity) returned a negative ratio and sorted as the emptiest under ascending consumers — the opposite of reality. Treat any non-positive capacity (full, or overfull mid-run after receiving volumes) as the fullest (+Inf) so it is a move source, never a target. Covered by negative-capacity and ordering tests. |
||
|
Chris LuandGitHub
|
d02ee6d5df |
balance: share replica-placement logic between shell and worker (#10169)
The replica-placement rule (data-center/rack/same-node limits plus host anti-affinity) existed three times: the shell's satisfyReplicaPlacement/isGoodMove used by volume.balance, fix.replication, and tier.move, and a line-for-line port in the maintenance balance worker. Move the canonical logic into weed/topology/balancer on a shared Location type; the shell and worker keep thin adapters that convert their own location representation and call it. Behavior is unchanged (the shared IsGoodMove keeps the shell's reject-move-to-self guard, and all four replica test suites pass). |
||
|
Chris LuandGitHub
|
bea1357d38 |
ec: skip physically near-full disks when placing EC shards (#10167)
EC placement scored destinations purely by free EC shard slots (derived from maxVolumeCount) and shard counts, blind to real disk fullness — the same defect as volume balancing. A disk that is physically full but still shows free EC slots kept being chosen, and EC shard bytes are captured by statfs free space yet not by any slot accounting, so the slot math is exactly the metric that can't see EC fullness. Treat a disk at/above 90% physical usage as having zero free EC slots at snapshot-build time, so every existing freeSlots>0 placement predicate excludes it. Applied in all three snapshot builders (shell countFreeShardSlots, the shared ecbalancer FromActiveTopology, and the worker ec_balance buildBalancerTopology) via the shared balancer.DiskTooFullAfter gate. Servers not reporting disk bytes fall back to slot-only behavior. ec.rebuild recovery is left ungated so shard recovery can still complete onto fuller disks. |
||
|
Chris LuandGitHub
|
77bf2a3ab0 |
volume.balance: gate on real physical disk usage (fixes #10160) (#10162)
* shell: add volume.balance -byDiskUsage to balance by actual data The default balancer ranks servers by slot density, dividing used volumes by MaxVolumeCount. When MaxVolumeCount is configured higher than the disk can hold, a physically near-full server looks nearly empty and gets picked as the move target, so balancing drains less-full servers onto an already-full one. -byDiskUsage ranks servers by the actual data they hold (sum of volume sizes) instead, so the fullest-by-data server is treated as full and balancing drains it. It assumes comparable disk sizes per disk type and still respects each server's free volume slots. Default behavior is unchanged. * plumb physical disk usage into topology, gate volume.balance on it Volume servers now report each disk's filesystem total/free bytes in the heartbeat, and the master stores them in DiskInfo. volume.balance uses them to skip any move target whose disk is already near full (-maxDiskUsagePercent, default 90), so an over-configured maxVolumeCount can no longer make a physically full server look empty and get drained onto. The gate judges each server against its own disk, so heterogeneous disk sizes are fine; servers that do not report bytes fall back to slot-only behavior. Rust seaweed-volume mirrors the heartbeat reporting. * admin: report real physical disk capacity when volume servers provide it The dashboard estimated server capacity as maxVolumeCount * volumeSizeLimit, which overstates it when maxVolumeCount is set higher than the disk holds. Prefer the filesystem capacity now reported per disk, falling back to the estimate for servers that do not report it. * worker: gate automatic balance on physical disk fullness too The maintenance balance worker selects the least slot-utilized server as the move destination, so an over-configured maxVolumeCount makes a physically full server look empty and get drained onto — the same defect as the shell command. Now that DiskInfo carries real disk bytes, skip any destination whose disk is at/above 90% used (per server, against its own disk); a full server can still be a source. When every candidate destination is full, create no tasks. Servers that do not report disk bytes are not gated. * balance: share the physical-disk-fullness gate between shell and worker The shell volume.balance command and the maintenance balance worker each grew their own copy of the disk-fullness gate (targetDiskTooFull / destinationDiskTooFull) and a maxDiskUsagePercent=90 constant. Pull both into weed/topology/balancer (DiskTooFullAfter + DefaultMaxDiskUsagePercent) so the policy has one home and the two balancers can't drift. * balance: harden the physical-disk gate Guard against a nil DiskInfo in the byte/slot lookups. Let a zero disk-capacity report clear previously stored bytes (0 means "not reported" for bytes, unlike maxVolumeCount), so a server that stops reporting falls back to slot-only instead of trusting stale capacity. In the worker, charge each planned move's bytes to its destination within a detection cycle so the gate sees a target fill up rather than only its heartbeat-time free space. Note the per-location capacity summing assumes one location per filesystem (the used ratio the gate relies on stays correct regardless; absolute capacity can over-report). |
||
|
Chris LuandGitHub
|
a9c0ed91b5 |
fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk (#10161)
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk DiskId 0 doubles as the first physical disk (Locations[0]) and the protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0 record onto the aggregate DiskId whenever that was non-zero, so on a multi-disk node the first disk's volumes merged into whichever disk held volumes[0]: the node reported one fewer disk, the sibling showed ~2x volumes, and per-disk max was smeared across the survivors. This surfaced as cluster.status and volume.list undercounting disks. Only treat 0 as unset when no record carries a non-zero DiskId; with a mix, 0 is a real disk and keeps its own entry. * fix(admin): resolve physical disk 0 in active-topology indexes rebuildIndexes re-derived each volume/EC record's physical disk id with the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so the two agreed only by sharing the bug. Now that SplitByPhysicalDisk keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and GetECShardLocations then matched no record and silently dropped every volume and EC shard on the first disk, starving balance and EC tasks. Build the indexes from the same SplitByPhysicalDisk reconstruction that builds at.disks, so the keys always resolve. One source of truth instead of a parallel normalize. * fix(ec): allow physical disk 0 as preferred EC shard target pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a valid physical disk and the uint32 zero value, so a best-scoring disk 0 was discarded and the non-matching fallback returned instead. Gate on bestScore. * test(admin): cover EC-shard index resolution for physical disk 0 rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC path too so a shard on disk 0 keeps resolving via GetECShardLocations. |
||
|
Chris LuandGitHub
|
a653a7f72a |
fix(shell): honor explicit fs.mergeVolumes from/to direction (#10159)
* fix(shell): honor explicit fs.mergeVolumes from/to direction
mergeVolumes only ever merged a smaller volume into a larger one. When the
user named both -fromVolumeId and -toVolumeId with the source larger than the
target, the planner produced an empty plan and the command printed just
"max volume size: N MB" and moved nothing.
Build the requested pair directly when both ids are given, instead of routing
through the size-descending heuristic. Read-only, empty, and wrong-collection
endpoints are rejected with a clear error rather than a silent no-op.
* fix(shell): allow fs.mergeVolumes into an empty target volume
Merging chunks into an empty volume is valid, e.g. consolidating data into a
freshly created or recently vacuumed volume. Only reject an empty source, which
has nothing to move.
* fix(shell): reject self-map in directed mergeVolumes planner
createMergePlan with from == to returned a {vid: vid} self-merge when called
directly. Guard it in the planner so it is correct independent of the Do
entrypoint.
|
||
 
|
b55a608ae0 |
feat: add collection pattern to delete empty volumes (#10129)
* feat: add collection pattern to delete empty volumes Co-authored-by: Codex <noreply@openai.com> * shell: match collection pattern with wildcard matcher Use wildcard.MatchesWildcard in the shared collection-pattern helper, matching command_volume_fix_replication's matchCollectionPattern. The flag only advertises '*' and '?', which is exactly what the matcher supports. --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
1e42dd77ca |
fix: avoid duplicate volume.list parent headers (#10126)
Co-authored-by: Codex <noreply@openai.com> |
||
|
|
378f9a64ff |
fix: apply collectionPattern during detection in volume.fix.replication (#10115)
* fix(shell): correct volume.list -writable filter unit and comparison * fix(shell): correct volume.list -writable filter unit and comparison * chore(shell): fix typo in EC shard helper param names * fix(shell): use exact match for volume.balance -racks/-nodes filter The old strings.Contains-based filter quietly included any id that was a substring of the user-supplied flag value (e.g. -racks=rack10 also matched rack1). Replace it with an exact-match set parsed from the comma-separated flag value, and add regression tests for both -racks and -nodes paths. Also fix a small typo in the "remote storage" error returned by maybeMoveOneVolume. * fix(shell): use exact match for volume.balance -racks/-nodes filter The old strings.Contains-based filter quietly included any id that was a substring of the user-supplied flag value (e.g. -racks=rack10 also matched rack1). Replace it with an exact-match set parsed from the comma-separated flag value, and add regression tests for both -racks and -nodes paths. Also fix a small typo in the "remote storage" error returned by maybeMoveOneVolume. * refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers * fix: apply collectionPattern during detection in volume.fix.replication * use existing wildcard.MatchesWildcard for collection matching It returns a plain bool, so drop the up-front filepath.Match validation and the path/filepath import that only existed to handle its error. * trim verbose comments to terse one-liners * drop redundant per-path collection guards Detection already filters by replicas[0].info.Collection. The repair guard re-checked pickOneReplicaToCopyFrom's collection (a different replica), so a mixed-collection volume could pass detection yet be skipped in repair without decrementing the counter, spinning the -apply loop. deleteOneVolume keeps its collectionIsMismatch safety. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
2efc0e1656 |
ec: recover EC shards whose .ecx index lives only on a peer server (#10108)
* ec: recover EC shards whose .ecx index lives only on a peer server A volume server that boots with EC shard files on disk but no .ecx index on any local disk cannot mount the shards, so the master never learns about them. ec.rebuild works off master-registered shards, so it sees the volume as short and gives up even though the shard data is intact. Add an operator-triggered recovery: VolumeEcShardsMount gains a recover_missing_index flag that makes the volume server fetch the missing .ecx (plus .ecj/.vif) from a peer holding it and mount the on-disk shards. ec.rebuild runs this across the cluster before planning, so orphaned shards register and the rebuild sees the true shard set. .ecx is an immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal that differs across holders, so the recovered node adopts the source peer's deletion view, like a balanced or rebuilt shard does. * ec: mirror missing-index recovery into the Rust volume server Port the #10104 recovery to seaweed-volume so the Rust volume server self-heals the same layout: EC shards on disk with the .ecx index only on a peer. Adds collect_ec_volumes_missing_index / mount_recovered_ec_shards to the store, recover_missing_ec_indexes (master LookupEcVolume + peer CopyFile fetch + mount) to the server, and the recover_missing_index flag on VolumeEcShardsMount. .ecx is the immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal, so the recovered node adopts the source peer's deletion view, matching the Go path. |
||
|
Chris LuandGitHub
|
95427b5573 |
security: add BearerPrefix constant for Authorization headers (#10101)
Introduce security.BearerPrefix ("Bearer ", RFC 6750) and use it
everywhere an "Authorization: Bearer <token>" header is constructed,
replacing the scattered "BEARER "/"Bearer " string literals. SeaweedFS
matches the scheme case-insensitively when parsing (security.GetJwt), so
behavior is unchanged; this removes the magic string and settles the
casing on the standard form. The parser's upper-case comparison stays as
is on purpose.
|
||
 Lisandro PinandGitHub
|
30f2dd5040 |
Weed shell ec.rebuild: Allow targeting rebuild to specific volume IDs. (#10087)
|
||
|
|
ddd11e44f9 |
feat: support marking volumes by collection (#9585)
* feat: add collection.mark shell command
Add collection.mark to mark all existing normal volume replicas in a collection as readonly or writable. The command runs in preview mode by default and requires -apply to execute changes. It reuses existing volume mark RPCs, supports default collection aliases, skips EC shards, and adds unit tests for option parsing and target collection logic.
* Revert "feat: add collection.mark shell command"
This reverts commit
|