1513 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandGitHub	47b491b53c	mount: version open file handles by filer log position (#10403) ... * filer: stamp a log position on lookup and remote-cache responses Metadata events are logged after their store write and stamped with the filer clock. Reading that clock before serving an entry therefore gives a timestamp with a causal guarantee: every event at or below it is reflected in the returned entry. Clients caching filer state can use it as the entry's version to order the response against subscription events, including events committed before the call but delivered after it. * mount: version open file handles by filer log position A subscription event refreshing an open handle did a second lookup; a transient failure left the handle pinned to its old entry with no retry, since the subscription cursor had already advanced. The deeper problem is ordering: the handle is a cache written by three unordered channels — the async invalidation worker, local mutation acks, and open-time lookups — and overwriting cached state safely requires knowing which write is newer. The filer log timestamp is that order, and it now travels with every value instead of being derived out of band. Events carry it natively; lookup and remote-cache responses carry the log position stamped before the serving read; mutation acks carry it in their returned event; and the local store pairs each read with a version cursor advanced under the same lock as the store write. Each handle records the version its entry reflects, and one rule replaces the per-site reasoning: state at or below the handle's version is old news and must not be installed. The invalidation itself applies the event's own entry — no lookup, so no transient-failure window — except under a cached parent, where the store entry is the ordered merge of the event and anything applied since, and its version outranks the event's. An uncached parent receives no store writes, so a hit there would be a stale leftover masking the event. A vacated path (delete, rename away) keeps the last entry so unlinked-but-open reads still work. Directory builds version the completed directory at the listing snapshot and re-invalidate buffered events at that version, since their mid-build refresh ran against an incomplete store. The tests replay every race this replaces machinery for: rollback of a newer local flush (queued, cached, and read-through), stale leftovers under uncached parents, the build window including abort, handles opened after an event was queued, events landing mid-lookup, and undelivered events at remote-cache time across a filer failover. * filer: serialize the log position fence with mutations, stamp mutation acks The fence stamped before an unlocked entry read could precede state the read returned: a mutation writes storage first and assigns its event timestamp only at notify time, so a lookup racing that window handed the mount an entry newer than its fence, and the event's later delivery looked like fresh news — destroying dirty pages for a change the handle already had. The mutation handlers already hold an exclusive per-path lock across read, write, and notify; the lookup and remote-cache reads now take it shared around the stamp and the read, making the fence exact: everything at or below it is in the entry, nothing above it is. A no-change update returns success without an event, leaving the mount nothing to fence with even though the response confirms current state. Create and update acks now carry a log position stamped under the same lock, and the mount falls back to it whenever the ack has no event. Also regenerate the VT marshalers, which the earlier generation missed: without them a VT round-trip silently zeroed every log position. * java: sync filer.proto * mount: scope store versions to what they vouch for; atomic handle install The store's version cursor claimed too much. Advanced by local mutation acks and directory listing snapshots, it inflated the version of store reads for unrelated paths whose events the subscription still owed, and those events were then fenced out permanently. The cursor now tracks subscription progress only — events arrive in log order, so everything at or below it has been delivered for every path — and a completed listing records its snapshot as a per-directory floor instead of a global claim. Local acks never touch it: they version their own handle directly. Buffered build events advance the cursor at delivery, since their store write may never happen (abort) while their invalidation is already queued; their read-through directory pairs no store read with it, and rename fragments are applied first. Concurrent first opens raced: a slower opener's older lookup could overwrite the newer entry a faster opener had installed, while the monotonic version kept the newer timestamp — an old entry fenced at a new version, immune to every correcting event. Entry and version are now installed as one decision under the handle map lock, and an install that does not outrank the handle's version is dropped. The remote-cache commit also escaped the fence: it wrote storage and notified without the path lock, so a lookup's shared-locked fence and read could land between the two and hand out the cached state under-versioned. The commit now re-reads and writes under the exclusive path lock, and backs off entirely when the entry changed during the download — the concurrent writer supersedes the cached content. * mount: floors gate store applies; installs respect handle users; renames join the fence A directory floor certifies the listing state as of its snapshot, but a delayed event at or below the floor was still applied to the store — rolling the content back to pre-snapshot state while the floor kept claiming the snapshot version, so the correcting events were fenced out of every future read. Events are now gated against the affected directory's floor, each half of a rename independently. Fences are lower bounds: a listing or lookup can include a mutation whose event has not been delivered yet, and that event later passes every gate carrying state the handle already holds. Such a re-delivery now advances the version without destroying dirty pages or reinstalling the entry — invalidating local writes over a no-op was the real damage in every remaining under-fence window, including the unlocked listing snapshot, which no per-path lock can serialize. The concurrent-open install moved from the map lock to the handle lock every reader, writer, and invalidation synchronizes on, and rejects what cannot improve the handle: dirty state (local writes would be lost), unversioned lookup responses (they cannot outrank anything, and two zero-version opens must not overwrite each other), and anything not strictly newer. New handles are still fully initialized before the map exposes them. Renames committed metadata and emitted events with no path lock, so a lookup could read the renamed state under a fence preceding its events. Both rename handlers now hold the source and destination locks, ordered by path, across commit and notification; descendants of a renamed directory are not individually locked and rely on the no-op re-delivery handling above. * mount: per-entry store versions replace the cursor and directory floors The store's aggregate versions — a global subscription cursor and per-directory listing floors — were versions at coarser granularity than the values they described, and every over-claiming bug in this series traced to that gap: an aggregate vouching for state its source never saw. Each store entry now carries the filer log position of the write that produced it — the event that applied it, or the listing snapshot that inserted it, recorded in the store's key-value space under the same lock as the entry write. The store becomes what the handle already is: a last-writer-wins register with one rule, install only what outranks the current claim. The cursor, the floors, their advancement rules, the pairing ordering constraint, and the floor gating all collapse into that rule. Applies are gated per entry, each half of a rename independently; an unversioned local write clears the claim its content no longer proves; version records lingering after a bulk folder wipe cannot fence a recreate, since a claim only blocks while its entry exists. Listing inserts are stamped at build completion, before the buffered replay so newer replayed events override the stamp. Filer side, the fence dance every versioned read must perform is now a single choke point, fencedFindEntry, so a future read RPC gets the lock-serialized stamp by construction rather than by convention. * mount: judge no-op re-deliveries against an immutable base, not the live entry The equal-state skip compared the incoming event to the live handle entry, but local writes mutate the live entry — size, timestamps, chunks — so a delayed event re-delivering the base the handle was opened with no longer matched, and the installer destroyed the dirty pages and rolled the entry back over nothing new. The handle now keeps an immutable snapshot of the filer state it last installed or acknowledged, refreshed at every install and mutation ack (flush acks snapshot the request entry before the id mapping mutates it), and the no-op judgment runs against that base: an event carrying the base brings nothing, whatever the live entry has diverged to since. * mount: tombstones for versioned deletes, absence floors, copy enrollment Four gaps in the per-entry version protocol, all the same shape: a versioned fact with nothing carrying its version. A deletion is a fact about a path with no entry left to hold it — clearing the record let a delayed older event resurrect the deleted path, permanently, since the deletion's own redelivery is dedup-suppressed. Versioned deletes now leave a tombstone record that fences without an entry; renames tombstone their source the same way. Plain records still only block while their entry exists, so records lingering after a bulk folder wipe cannot fence a recreate. A completed listing proves absences as well as presences: a name it omitted was deleted as of the snapshot, and a delayed create below the snapshot re-creates it. The snapshot is kept per directory strictly as an absence fence, consulted only when a path has neither an entry nor a version record — present entries carry their own versions and never touch it, which is what separates this from the over-claiming floor it replaces. A rebuild against a pre-upgrade filer returns no snapshot; stamping now clears the children's records in that case, so a reinserted entry cannot reactivate the stale claim its previous incarnation left behind and reject valid events below it. Server-side copies installed the copied entry without enrolling in the base protocol, so the copy's own event differed from the stale pre-copy base and destroyed writes made to the destination after the copy. The install now refreshes the base and takes its version from the fenced readback. * mount: deletion facts outlive the cache's knowledge of the entry A versioned delete of a path the store held no entry for recorded nothing, so a delayed older event recreated the path — permanently, with the deletion's redelivery dedup-suppressed. The tombstone is now written whenever a versioned event vacates a path: the deletion is a fact about the path, not about what this cache happened to hold. For an absent entry, the listing's absence floor now speaks whatever older record remains: a tombstone at one position does not exhaust what is known about the path when a newer snapshot has confirmed the name still absent, and an event between the two was slipping past both. A committed copy whose readback failed installed a synthesized base with local timestamps; the copy's real event legitimately differs from it, and was read as foreign state — destroying writes made to the destination after the copy. The handle now marks that its own event is en route and adopts that event's state as the base without touching the live entry or the dirty pages; the adoption is one-shot, so a genuinely foreign event still invalidates. * mount: authoritative acks cancel pending event adoption; tombstones scoped and pruned The copy-event adoption flag could outlive its purpose: a flush after the failed readback installs a newer base and advances the version, the copy's own event is then version gated without consuming the flag, and the next genuinely foreign event was silently adopted — base advanced, live entry and dirty pages untouched — leaving the mount to later overwrite that remote change. Every local acknowledgment now installs its base through one helper that also cancels any pending adoption: the ack supersedes the mutation the adoption was waiting for. Tombstones were written for every versioned delete under the mount and survived directory eviction by design, growing LevelDB with historical deletions on delete-heavy mounts. They are now scoped to directories whose cached state the fence actually protects — an uncached parent never serves from the store nor applies the resurrecting insert — and a completed listing prunes the direct-child tombstones its absence floor supersedes, leaving only those above the snapshot. The store gains a key-prefix visitor for the sweep. * mount: acked saves install their value; trailer snapshots; direct-child prune range A version must never advance without its value. saveEntry stamped any open handle with the acknowledgment's version, but a handle opened while the save was in flight holds the pre-mutation entry — stamping it fenced out the events carrying the state it lacked, permanently, with the local apply performing no invalidation and the redelivery deduplicated. The acknowledged entry is now installed together with its version, through the same guarded install the racing-open path uses: under the handle lock, only when it outranks the handle, never over dirty local writes. Empty listings return no in-band snapshot — a snapshot-only response would be read as an entry by older consumers — so directories that end empty gained no absence floor and their tombstones were never pruned. The filer now sends the snapshot in the stream trailer, which older clients ignore, and the client reads it when no in-band snapshot arrived. Empty directories get real floors, their tombstones prune, and their buffered replays gain the snapshot filter instead of the replay-all fallback. Version records now encode the parent directory and name separated by a NUL, making a directory's direct children one contiguous key range: the tombstone prune scans exactly them under the cache lock, instead of walking every descendant record — the whole store, for root. * mount: fix dirty-page loss, uid/gid base, download race, copy adopt, leak; dedup Correctness fixes from the versioned-invalidation review: - A foreign delete/rename-away of a file held open with unflushed local writes destroyed the dirty pages unconditionally. A process may keep writing to an unlinked-but-open file and those writes were already acknowledged; preserve the pages when the handle is dirty. - downloadRemoteEntry stored the handle's base with filer-side uid/gid while every candidate it is later compared against is in local form, so under a non-identity UidGidMapper an unchanged re-delivery looked foreign and force-destroyed dirty pages. Map the base to local. - downloadRemoteEntry wrote the entry/base/version triple under only the handle's shared lock, so two concurrent reads of the same remote-only file could tear it. Serialize the install with a dedicated mutex (invalidation is already excluded by the exclusive handle lock). - A committed server-side copy whose readback failed adopted the FIRST event past the version gate as its base; a foreign write delivered first was silently swallowed. Adopt only an event whose content matches the synthesized base — the copy's own event — and install any other normally. - The deferred-create path relied on AcquireFileHandle installing the passed entry on a pre-existing handle, which the version rework dropped. Restore that install in the compat wrapper; the versioned open path keeps its gated install. Growth and hot-path cost: - Per-entry version records and tombstones leaked when a directory was evicted or read-through without a rebuild. An uncached directory gates its own inserts, so its records fence nothing; clear a directory's child version records when it is wiped for eviction. - FindEntry paid for the version KvGet on every lookup/getattr cache hit and threw it away. FindEntry now reads only the entry; the hot lookupEntry cache-hit path skips the version entirely. Cleanups: - Extract ackVersionTsNs over the shared response interface, replacing the metadata-event-else-log-ts snippet copy-pasted at four ack sites. - Extract acquireRenamePathLocks, replacing the verbatim sorted two-path lock fence in both rename handlers. * mount: no resurrection on foreign delete, version no-event acks, gate downloads, tighten copy adopt Follow-ups to the review patches: - Preserving dirty pages on a foreign delete let the next flush pass the isDeleted guard and CreateEntry, resurrecting the remotely-unlinked name. Mark the handle deleted in the vacate branch: the open fd can still read its buffered writes, but a flush no longer recreates the file. - A no-event acknowledgment (log fence only) synthesized a metadata event with TsNs 0, so the cache stored the entry unversioned and an older subscriber event rolled it back. Stamp the synthesized event with the ack's log position at all four ack sites. - downloadRemoteEntry serialized its install but did not check the version, so an older response arriving last overwrote the entry/base while the monotonic version kept the newer value, fencing corrections out. Install only when the response is at least as new as the handle. - sameEntryContent compared only size and chunks, so a foreign chmod with unchanged content was adopted as the copy's own event. Compare everything except server-assigned timestamps, so a metadata-only foreign change installs instead. * mount: trim comments to the non-obvious why The versioning work accumulated multi-line comment blocks restating what the code says. Keep the constraint a reader cannot derive — why a fence is exact, why a version must not advance without its value, why an uncached parent's records fence nothing — and drop the rest. * mount: distinguish rename from delete, tighten the download and adopt gates - A rename emits a nil old-path invalidation just like an unlink, so the vacate branch marked the handle deleted and later writes through the already-open descriptor were skipped instead of persisted. Carry the delete/rename distinction on the invalidation and mark only an actual delete. - The remote-download install accepted an unversioned response regardless of the handle's version, so during a rolling upgrade a delayed response could install stale content under a newer version. Require the response to be at least as new, with one exception: a handle still lacking local chunks takes the content anyway — it cannot read without it — but does not claim the response's log position. - Copy-event adoption returned without installing, so a foreign touch arriving before the copy's own event lost its timestamps. Content is unchanged either way, so the dirty pages stay valid; a clean handle now takes the entry, while a dirty one keeps its diverged version. * mount: one directory floor instead of a record per child; agree on TTL Review feedback: - Build completion wrote one KV record per direct child inside the cache write lock, so a large directory stalled every other cache operation for O(children) store writes. The directory's listing snapshot already covers every child it saw; make that floor the version for any child without a record of its own, and a child earns a record only when a later event touches it. One map write per build replaces the per-child writes, with the same fencing. - The presence probe read the store directly and so counted a TTL-expired entry as present, judging the path by a record describing content that has logically vanished. It now applies the same expiry the read path does, and an expired path falls back to its directory floor. - Preserve ErrNotFound identity when the commit-time re-read finds the object deleted, so callers still surface a 404. - Assert the rename-away source fence timestamp in the invalidation test. Also record the tombstone ceiling: distinct deleted names in a cached directory accumulate until it is rebuilt or evicted, which prunes everything at or below the new snapshot. * mount: pin the fence's clock domain instead of letting skew decide A log-position fence is stamped by one filer's clock under that filer's in-process lock, so comparing it to an event another filer logged is comparing two unrelated clocks. The two error directions are not equally costly: applying an event the fence already covered is a re-apply the base-equality check absorbs, while skipping one it does not cover leaves the handle holding exactly the state the event was meant to correct, with the subscription cursor already past it — the unhealable staleness this whole PR exists to remove. So refuse to guess. Fences now carry the signature of the filer that stamped them, and a handle records it alongside the position. An event is only fenced out when the filer that logged it is the one that stamped the fence — the logging filer appends its own signature, so its presence identifies the clock domain. Events from any other filer are applied. Positions taken from events keep comparing as before; the subscription already delivers those in order. The invalidation callback takes a struct now: it carries the path, entry, position, delete/rename distinction, and signatures, and was about to need a fifth positional parameter. * mount: follow a foreign rename; key page invalidation on content, not equality - A rename's old-path invalidation now carries the destination, and the handle follows the file there: an open fd tracks the inode, and leaving it on the old path made its next flush recreate that name instead of updating the renamed file. - Dirty pages overlay content, so only a content change invalidates them. Keying that on exact equality meant any timestamp-only event destroyed them, which the copy-adoption marker existed to paper over — a foreign touch could consume the marker and leave the copy's own event to drop the post-copy writes. Comparing content instead makes the marker unnecessary, so it is gone: a metadata-only event keeps the overlay, and a dirty handle keeps its diverged entry unless foreign content supersedes it. - A remote download response that is merely older is now refused even when the handle still lacks chunks; only an unversioned one is taken (and claims no position), since an older response's content predates what the handle reflects. - A refused or unversioned download no longer publishes to the metadata cache, where a zero-position event would clear the entry's version and let an older subscriber event roll the cache back. * mount: page invalidation keys on content alone; unversioned writes claim no position - sameEntryContent compared everything but timestamps, so a foreign chmod, chown, or xattr change counted as a content change and destroyed the dirty-page overlay. It was strict only to serve the copy-adoption marker, which is gone; its one caller now asks the question it actually needs — did the bytes change — so metadata-only events leave the overlay alone. - A rename over an existing file destroys that file, but its open handle was left live and still pointed at the name the renamed source now occupies, so its flush could overwrite it. MovePath already reports the displaced inode; mark that handle deleted. - An acknowledgment was refused whenever its position was numerically lower, even when a different filer stamped the fence it lost to. Two known, differing signatures mean unrelated clocks, so the comparison no longer applies there; unknown signatures still compare as before. - A local write with no log position behind it now records that explicitly instead of deleting its version record. Absence means the directory listing covers the path, which is why the snapshot floor applies; local content the listing never saw must not inherit it, or the events that would correct it are fenced out. * mount: widen the existing lookup functions instead of forking WithVersion twins The versioning work grew a parallel function for every accessor that needed to return a log position — lookupEntryWithVersion beside lookupEntry, maybeLoadEntryWithVersion beside maybeLoadEntry, FindEntryWithVersion beside FindEntry, AcquireFileHandleWithVersion beside AcquireFileHandle, advanceEntryVersion beside advanceEntryVersionTsNs, plus a getPbEntryWithVersion wrapper and an InsertListedEntriesForTest hook. Two names for one operation is two places to keep in step, and the split let callers pick the one that happened to compile. Each pair is now the single original name carrying the position, with callers that do not want it discarding it. filer_pb.GetEntry returns the fence its response already carried rather than a mount-side wrapper re-issuing the lookup, and InsertEntry takes the position its content reflects rather than a test-only twin that inserted without one. The one behavioural knot the merge exposed: AcquireFileHandle had been installing the entry on a pre-existing handle only in its unversioned form, which conflated 'the caller is authoritative' with 'the lookup had no version'. Deferred create is the only caller that means the former, so it now installs explicitly and the map function just acquires.	2026-07-23 17:44:02 -07:00
3 people	490379bff3	Add codespell support with configuration and typo fixes (#10393) ... * Add GitHub Actions workflow for codespell on master * Add rudimentary codespell config * Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers like allLocations, publishErr, ReadInside, FlushInterval. Also skip templ-generated *_templ.go files, and whitelist a handful of short/domain-specific words (visibles, fo, te, ser, bject, unparseable, keep-alives, tread, anc, ue) that show up as false positives across the tree. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix ambiguous typos and protect false positives Fixes typos that codespell reports with multiple candidate suggestions (so `codespell -w` cannot auto-apply them), plus one inline pragma and one config entry to protect legitimate identifiers. Manual fixes (single correct answer chosen from context): - pattens -> patterns (5x) in filer/upload/shell flag help strings - finded -> found (2x) in tarantool storage.lua comment - spacify -> specify (2x) in helm chart values.yaml comment - wether -> whether in skiplist.go docstring - simpe -> simple in mq schema test case name False-positive protection: - Add `//codespell:ignore` next to `source GET's` (possessive of HTTP verb) in s3api_object_handlers_copy_stream.go - Whitelist `auther` in .codespellrc — it's a local variable meaning "authenticator" in weed/security/tls.go, not a typo of "author". Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Extend codespell ignore list: .git-meta path and thirdparty groupId Also skip `.git-meta` (scratch dir for commit messages that may contain typo words verbatim) and whitelist `thirdparty` — it appears as the literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms and cannot be renamed. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w Auto-applied fixes to the 44 remaining single-suggestion typos across docs, comments, log messages, tests, config, and one Java pom. === Do not change lines below === { "chain": [], "cmd": "uvx codespell -w", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^ * Revert breaking codespell fixes; whitelist unknwon and atleast Two of the auto-applied `codespell -w` fixes were false positives that would break the build/tests: - go.mod: `github.com/unknwon/goconfig` is a real Go module path — the upstream author's GitHub handle is literally `unknwon`. Renaming to `unknown` would fail dependency resolution. - test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}: `atleast` is a literal CLI mode value (a string constant compared and passed as a positional argument). Rewriting to `at least` splits it into two arguments and breaks the mode check. Reverted those files and whitelisted both words in .codespellrc so future runs won't re-suggest the same broken fixes. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-07-22 14:38:06 -07:00
5 people	6c4eb95a3a	fix(admin): implement ApplyPluginConfigFromToml to propagate settings (#10388) ... * fix(admin): implement ApplyPluginConfigFromToml to propagate settings to plugin config store * Update weed/admin/dash/config_toml.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * admin: overlay admin.toml onto plugin configs through bootstrap defaults Creating a config from scratch at startup skipped the descriptor-defaults bootstrap, so a job type with only worker keys in admin.toml persisted Enabled=false and RetryLimit=0 and silently stopped running. Overlay existing configs at startup, and apply the same overlay in enrichConfigDefaults when the plugin bootstraps a fresh config from descriptor defaults. Also place collection_filter in the admin values where workers read it, map preferred_tags as a string list, and stamp UpdatedAt. * admin: trim the admin.toml help text and call-site comment * admin: clamp toml retry values to the int32 range * admin: fail startup when admin.toml cannot reach the plugin config The legacy overlay already aborts startup when declared settings cannot persist; continuing here would let workers bootstrap with stale values. * admin: fix the retry clamp test on 32-bit A 32-bit int cannot hold the oversized toml value, so viper returns 0 before the clamp runs. --------- Co-authored-by: baracudaz <baracudaz@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-07-21 14:00:37 -07:00
Chris LuandGitHub	5a54beac80	EC decode: read shards with the encode-time block layout (#10385) ... * erasure_coding: WriteDatFile takes the encode-time dat size for the shard block layout * volume server: derive EC decode layout from the encode-time dat size, not the live extent * erasure_coding: test decode after tail deletions shrink the live extent below a large-block row * seaweed-volume: write_dat_file_from_shards takes the encode-time dat size for the shard block layout * seaweed-volume: derive EC decode layout from the encode-time dat size, not the live extent * seaweed-volume: test decode after tail deletions shrink the live extent below a large-block row * erasure_coding: reject decoding with no data shards * worker: record the encode-time dat size in the .vif * erasure_coding: fall back to the shard-derived layout only when the encode-time dat size is missing * erasure_coding: reject an ambiguous shard-derived block layout * seaweed-volume: fall back to the shard-derived layout only when the encode-time dat size is missing * seaweed-volume: reject an ambiguous shard-derived block layout	2026-07-21 08:59:14 -07:00
Chris LuandGitHub	76ec1d8f0f	s3: accept raw semicolons in query strings (#10305) ... * s3: accept raw semicolons in query strings Go's url.ParseQuery drops any key=value pair containing a raw ';'. A presigned PUT that signs content-type carries X-Amz-SignedHeaders=content-type%3Bhost; when a client or proxy decodes the %3B, the parameter vanished and the upload failed with MissingFields, while AWS accepts the raw ';' as query data. Re-encode it before routing so the pair survives parsing and signature verification. * iam, iceberg: recover raw-semicolon query pairs on the other listeners The standalone IAM API verifies SigV4 with a canonical query recomputed from the parsed query, and Iceberg REST warehouse/parent values may legally contain ';'. Move the normalization middleware to util/http and attach it to both routers.	2026-07-10 12:17:35 -07:00
3 people	c17aebeecf	fix(filer.backup): prevent silent backup data loss on transient not-found (#10295) ... * fix(filer.backup): stop silently dropping events on transient not-found Under a write burst, filer.backup could consume a metadata event (advancing the persisted offset) without replicating the file, with no error logged: 1. filersink CreateEntry/UpdateEntry swallowed replicateChunks errors (glog.Warningf + return nil), so the offset advanced past entries that were never written. 2. The manifest-chunk branch of replicateChunks resolved via LookupFileId with no retry, unlike the data-chunk branch — transient lookup races dropped exactly the large manifest-backed files while small inline-content siblings landed. 3. isIgnorable404 matched "LookupFileId" / "volume id ... not found", misclassifying those races as genuine source 404s at the backup layer. Fix: on a replicateChunks failure the filer sink now skips only when the live source has moved past the replayed version (deleted or strictly-newer mtime) — lossless, a later event carries the current content — and propagates otherwise so the event is retried. The manifest resolve retries transient errors like the data-chunk path. isIgnorable404 is narrowed to genuine 404s; non-filer sinks and the initial-snapshot walk, which relied on the broad match as their only lossless-skip valve, now make the same live-source decision (filersink.SourceSupersedes) instead of retrying forever on a permanently gone volume. Tests cover propagation of unconfirmed lookup failures, the narrowed 404 classification, and the supersession guards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filer.backup): derive supersession path with directory fallback eventSourceSuperseded built the source path from NewParentPath alone. Legacy metadata events (persisted by older filers) carry an empty NewParentPath, so the probe looked up "/<name>", read the miss as "source gone", and skipped a live file on a transient lookup error — the silent drop this change is meant to eliminate. Derive the path via MetadataEventTargetFullPath (the same directory fallback genProcessFunction uses) and cover both event shapes with TestEventSupersessionProbe_PathDerivation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filersink): retry manifest resolve only for transient errors, bounded when unverifiable The manifest-resolve retry stopped only when hasSourceNewerVersion proved the source moved past the replayed version, which wedged the sink in two cases: incremental sinks use dated target keys that cannot map back to a source path (supersession never provable), and permanent resolve errors (corrupt manifest data, bad file ids) fail forever while the source entry stays live. Gate the retry instead: keep retrying only transient errors (volume-lookup races, network interruptions), stop after a few attempts when supersession cannot be checked, and propagate everything else immediately so the configured metadata error policy applies (-disableErrorRetry included). Propagation is lossless: filer.backup's fallback decides with the event's real source key, and both filer.backup and filer.sync re-deliver the event (RetryForeverOnError) without advancing the offset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filer.backup): make error classifiers nil-safe isIgnorable404, isSourceLookupError, and isTransientResolveError called err.Error() without a nil guard. All current call sites pass a non-nil error, but the guard is free and matches isRetryableNetworkError. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-10 01:47:46 -07:00
3 people	9d75048594	admin: respect filerGroup for cluster discovery (#10170) ... * Respect filerGroup in admin discovery Admin discovery previously queried master cluster nodes with an empty filer group, so filers registered under a non-default group could not appear in the admin UI. Add an admin filerGroup flag and carry it through cluster-node discovery requests while preserving the empty default behavior. Constraint: SeaweedFS master ListClusterNodes filters by exact filer_group. Rejected: Discover all groups implicitly | no existing admin or shell behavior exposes cross-group discovery. Confidence: high Scope-risk: narrow Directive: Keep admin cluster discovery scoped to the configured filerGroup unless an explicit all-groups API is added. Tested: docker run --rm -v "$PWD:/src" -w /src golang:1.25 go test ./weed/admin/dash -run TestListClusterNodesRequest -count=1 Tested: docker run --rm -v "$PWD:/src" -w /src golang:1.25 go test ./weed/command -run '^$' -count=1 Not-tested: full repository test suite * mini: pass filer group to admin cluster discovery miniAdminOptions.filerGroup was never initialized, so startAdminServer dereferenced a nil *string. Share the filer.filerGroup flag pointer so the co-located admin queries the same group the filer registers under. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-07-01 10:56:58 -07:00
3 people	ebeab4b6ec	feat(filer.sync.verify): reclassify chunk-slice-order ETag diffs as CHUNK_REORDER (#10177) ... * feat(filer.sync.verify): reclassify chunk-slice-order ETag diffs as CHUNK_REORDER filer.ETagChunks concatenates per-chunk MD5s in stored slice order without normalising by offset, so byte-identical content written by two different paths (S3 multipart part-completion order on the source vs filer.backup replication arrival order on the destination) yields different file ETags. filer.sync.verify reported these as ETAG_MISMATCH even though the files are equal. Add a second-pass check on every ETAG_MISMATCH: when both sides derive their ETag from chunks (no attr.Md5) and hold a manifest-free, non-overlapping chunk set that, once sorted by offset, matches element-wise on (offset, size, ETag), classify the file as CHUNK_REORDER. Such files are content-equal, so they are not counted as errors and do not affect the exit code; they are listed only at higher verbosity (weed -v=1), while the summary always shows their count. The check stays conservative: a stored attr.Md5 (order-independent content hash), a differing chunk count, an overlapping/duplicate offset (whose visible bytes are resolved by timestamp), or a manifest chunk all remain ETAG_MISMATCH. * filer.sync.verify: decline chunk-reorder fast path on empty per-chunk ETag An empty or undecodable per-chunk ETag is not a content fingerprint, so element-wise (offset, size, ETag) equality can't prove the bytes match. Treating "" == "" as content-equal could reclassify a genuine divergence as CHUNK_REORDER and drop it from the error count. Decline such chunk sets so they stay ETAG_MISMATCH. * filer.sync.verify: emit CHUNK_REORDER in JSON output regardless of -v The -v=1 gate belongs to the human text report only. Applying it before the jsonOutput branch dropped the per-file CHUNK_REORDER records from NDJSON while the summary still counted them, so a machine consumer saw a non-zero count with no records to reconcile it. Gate the text path only; JSON always emits. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-07-01 10:38:21 -07:00
4 people	bdcc3154ed	refactor: centralize genUploadUrl in UploadOption (#10164) ... * refactor: centralize genUploadUrl in UploadOption Replace inline genFileUrlFn closures with operation.GenUploadUrl field: - Add GenUploadUrl func(host, fileId) string to UploadOption struct - Add GenUploadUrlProxy(filerAddress string) utility function - Remove genFileUrlFn parameter from UploadWithRetry signature - Update all callers: mount, gateway, mq, filer_copy, filer_sync This matches the weed mount -filerProxy pattern exactly, factorizing the URL generation logic across all consumers. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * docker release: run all platform jobs in one wave, cache rocksdb compile Drop max-parallel so the 13 per-platform builds run together instead of two waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late). Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is sha-independent, so it caches across releases and stops being the ~16-min long-pole that gates the merge fan-in. go-build variants stay mode=min. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * refactor: centralize genUploadUrl in UploadOption Replace inline genFileUrlFn closures with operation.GenUploadUrl field: - Add GenUploadUrl func(host, fileId) string to UploadOption struct - Add GenUploadUrlProxy(filerAddress string) utility function - Remove genFileUrlFn parameter from UploadWithRetry signature - Update all callers: mount, gateway, mq, filer_copy, filer_sync This matches the weed mount -filerProxy pattern exactly, factorizing the URL generation logic across all consumers. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * Remove accidental ROCmFPX submodule reference * gofmt chunk upload option block * Preserve broker cipher and re-read proxy filer per upload attempt Chunk uploads must keep the configured Cipher, and both the mount and broker current filer can change on failover, so build the proxy upload URL inside the closure instead of capturing the address once. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-06-30 20:45:43 -07:00
Chris LuandGitHub	7c3c5ed2a4	fix(filer.sync.verify): sort listings client-side before merge (#10117) ... * fix(filer.sync.verify): sort listings client-side before merge The merge walks both filers' directory listings in lockstep and needs them in the same byte order. A filer before 4.32 with a locale SQL collation lists case-insensitively while a 4.32+ peer lists byte-ordered, so comparing two such clusters returns the same names in a different order and the merge desyncs into spurious MISSING / ONLY_IN_B. Buffer and sort each directory client-side so both sides agree on order regardless of filer version or store backend. Trades the streaming source's O(buffer) memory for O(directory) per side, fine for a one-shot verify CLI; both sides still load concurrently. Claude-Session: https://claude.ai/code/session_01BKsBdKYFNCEjeHLjJfumPF * fix(filer.sync.verify): surface listing errors before merging A listing that fails mid-stream leaves a partial, unsorted buffer. Now that both sides are fully buffered anyway, check each side's error right after the loads finish and before the merge, so partial entries can't emit spurious MISSING / ONLY_IN_B before the error aborts the run. Claude-Session: https://claude.ai/code/session_01BKsBdKYFNCEjeHLjJfumPF	2026-06-26 10:27:18 -07:00
Chris LuandGitHub	5112da98a2	mount: skip redundant permission checks under default_permissions (#10089) ... With default_permissions (the mount default) the kernel enforces unix permission bits from the getattr/lookup attributes before it ever calls Open, Create, or Mknod. The mount was re-checking permissions in AcquireHandle and createRegularFile anyway, which duplicated the kernel's work and kept the supplementary-group lookup on the per-file hot path. Gate only the mode-bit access check on default_permissions being off, so a non-root copy does no permission work on open/create. createRegularFile still loads the parent to validate it exists, since the create RPC skips the filer-side parent check. With default_permissions off the mount remains the sole enforcer, so the full check still runs.	2026-06-24 14:24:51 -07:00
4 people	e1f89f85f2	fix(filer): apply -filer.disk default to metadata log assigns (#10080) ... * fix(filer): apply -filer.disk default to metadata log assigns Metadata event log writes call operation.Assign directly and used only FilerConf path rule DiskType. When filer.conf rules were missing or unmatched, the master received an empty DiskType and grew volumes on the built-in hdd layout. Mirror resolveAssignStorageOption: wire FilerOption.DiskType into the Filer, fall back when the matched path rule has no disk type, and return the matched rule from resolveMetadataLogAssignDiskType to avoid duplicate MatchStorageRule lookups. Co-authored-by: Cursor <cursoragent@cursor.com> * mini: fall back to -volume.disk for filer default disk type weed server copies -volume.disk into the filer disk default when -filer.disk is unset; weed mini did not, so metadata-log assigns sent an empty disk type on clusters that only tag volumes (e.g. hot/warm). --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-06-24 10:47:11 -07:00
3 people	091d953c34	fix(benchmark): close CPU profile file handle after profiling (#10048) ... Co-authored-by: Contributor <contributor@example.com>	2026-06-22 20:33:22 -07:00
Chris LuandGitHub	53342c9ba6	mini: resolve admin credentials from security.toml and env vars (#10021) ... * mini: resolve admin credentials from security.toml and env vars weed mini started the admin UI without resolving admin.user/admin.password (and the read-only pair) from security.toml [admin] or WEED_ADMIN_* env vars, so the only way to protect the UI was the -admin.password flag. The standalone weed admin command applies these fallbacks in runAdmin via applyViperFallback; the mini path calls startAdminServer directly and skipped it, leaving authRequired false and the UI unauthenticated. * mini: load admin.toml maintenance settings The mini admin path runs ApplyMaintenanceConfigFromToml (via startAdminServer) against the global viper, but runMini never merged admin.toml, so file-based maintenance task settings ([maintenance.vacuum], .balance, .erasure_coding) were ignored under mini while the standalone weed admin honored them. Load it alongside master/volume config. * mini: support -admin.urlPrefix for the admin UI Expose the reverse-proxy subdirectory prefix that the standalone weed admin already supports, so the mini admin UI can run under e.g. /seaweedfs. The prefix is normalized the same way and passed through to startAdminServer.	2026-06-19 13:04:04 -07:00
Chris Lu	443b5b1184	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2026-06-17 13:45:47 -07:00
Chris Lu	0c343e76eb	admin: don't log normal 2xx/3xx HTTP requests (incl. 304 cache hits)	2026-06-17 11:34:38 -07:00
Chris LuandGitHub	e411ff491d	volume: remove ec.bitrotChecksum and ec.bitrotBlockSizeMB flags (#10000) ... EC bitrot protection is now a fixed default: always on at 16 MiB block granularity. These volume-server flags exposed needless configurability; the package defaults in erasure_coding (BitrotProtectionEnabled, BitrotBlockSize) are retained and still drive sidecar generation. Drop the now-unused erasure_coding import.	2026-06-17 00:07:38 -07:00
Chris LuandGitHub	5e8152b81c	storage: register tier backends at the binary composition root (#9989) ... The s3 and rclone tiered-storage backends were registered via blank imports in weed/storage (volume_tier.go and volume_info/volume_info.go). That forced every library consumer of weed/storage -- weed/shell, and through it external tools -- to link aws-sdk-go and, under the rclone build tag, the full rclone backend set and its cloud-storage SDKs, even though those consumers never tier volumes. Move the registrations into a new weed/storage/backend/all aggregator and blank-import it once from weed/command, the binary's composition root. The weed binary still registers both backends; weed/storage and its library consumers no longer pull the backend SDKs into their dependency graph.	2026-06-16 11:47:32 -07:00
Chris LuandGitHub	9266aaa88e	security.toml: document WEED_ env override for jwt signing keys (#9981) ... security.toml: document WEED_ env override for the jwt signing keys These keys are HMAC secrets; spell out the env-var mapping so they can be injected from a secret store instead of living in the config file.	2026-06-15 13:25:06 -07:00
4 people	3bf3d29058	fix(command): preserve fuse option after writers (#9972) ... * fix(command): preserve fuse option after writers Problem: FUSE mount option parsing skipped the option immediately following concurrentWriters, so values such as concurrentReaders could be silently ignored. Root cause: runFuse incremented the options loop index inside the concurrentWriters case in addition to the loop increment. Co-authored-by: Codex <noreply@openai.com> * test: save and restore mountOptions pointers, not their values The fields are reassigned to fresh heap variables during runFuse, so dereferencing to back up/restore mutated throwaways instead of the flag-bound originals and could nil-panic on unset fields. --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-06-15 13:11:35 -07:00
Chris LuandGitHub	37962e2445	admin: configure maintenance tasks via admin.toml (#9926) ... * admin: configure maintenance tasks via admin.toml Maintenance task settings could only be edited in the admin UI and live under <dataDir>/conf, so they silently reverted to defaults whenever the data directory was recreated. An optional admin.toml now declares vacuum, balance, and erasure coding settings; keys set there are written through to the persisted task configs at every startup, overriding UI edits, so the configuration stays declarative. Generate an example with "weed scaffold -config=admin". * vacuum: round min volume age up to whole hours MinVolumeAgeSeconds was truncated by integer division when converted to the hour-granular protobuf field, so a sub-hour setting silently became 0 and disabled the age guard. * admin: split and normalize preferred_tags from admin.toml A comma-separated string, as set via environment variable, came through viper as a single slice element. Split on commas and reuse util.NormalizeTagList, matching the plugin config path. * scaffold: clarify admin.toml wording	2026-06-11 11:04:52 -07:00
Chris LuandGitHub	e56a1c4c05	admin: pre-gzip embedded static assets, add cache headers (#9918) ... The admin UI served embedded static files uncompressed and without cache headers: embed.FS has zero mod times, so no Last-Modified, no ETag, no 304s -- every page load re-downloaded ~700KB of css/js in full, which gets painful over slow or tunneled links. Gzip the static tree at generation time (go generate ./weed/admin) and embed only the compressed mirror, shrinking the binary ~1.5MB. The handler hands the pre-compressed bytes to gzip-capable clients, decompresses for the rest, and sets Cache-Control, per-variant content-hash ETags and Vary so repeat loads revalidate with a 304. bootstrap.min.css goes 232KB -> 30KB on the wire. A drift test keeps static_gz/ in sync with static/.	2026-06-10 12:54:36 -07:00
Chris LuandGitHub	2ac5aa72c7	add elastic8 filer store for Elasticsearch 8 (#9916) ... * elastic: fix listing against a missing or empty directory index The refresh 404 leaked into the named return, so the first listing of a directory whose index does not exist yet returned an error instead of an empty result. Sorting also fails on an index with no documents ("No mapping found for [_id] in order to sort on"); unmapped_type keeps the resumed-listing path working there. * add elastic8 filer store for Elasticsearch 8 Elasticsearch 8 disables _id fielddata by default, so the elastic7 store's directory listings fail with "Fielddata access on the _id field is disallowed". elastic8 uses the same client and configuration options, but also indexes the document id as an Id field and sorts listings on Id.keyword.	2026-06-10 12:10:49 -07:00
Chris LuandGitHub	e12052ee6b	fix(filer.sync): replicate a rename as an atomic move, not a no-op update (#9895) ... * fix(filer.sync): replicate a rename as create-then-delete, not an in-place update A rename arrives as a single metadata event carrying both the old and new entry. The filer sink was routed to UpdateEntry, which looks up the old path but issues the update against the new parent without changing the name — and the filer UpdateEntry RPC cannot move an entry. So the rename was dropped: the old path lingered and the new path never appeared (same-dir renames rewrote the old name in place). Route a real move (the sink path changed) through CreateEntry(new) then DeleteEntry(old) in both the replicator and the filer.sync/backup driver, the way the other sinks already handle it; reach UpdateEntry only for true in-place updates. Create before delete so a crash between the two leaves the entry visible rather than lost. * fix(filer.sync): derive the rename delete key like the create key, guard the watched root The rename delete leg rebuilt the old key with a raw util.Join, bypassing the sink-side key normalization the create leg gets from buildKey — so a rename could create the new entry and then fail to delete the old one under a transformed key. Build the old key through buildKey too, and skip the delete when the moved entry is the watched root itself (where the old key would resolve to the target root and recursively delete the whole sink tree). * test(filer.sync): cover the in-place update delete-then-create fallback order The recording sinks always reported foundExisting, so the fallback that an in-place update takes when the entry is missing on the sink was never run. Make it configurable and assert the fallback deletes before it recreates the same key, in both the replicator and the filer.sync drivers. * feat(filer.sync): move filer-sink renames natively via AtomicRenameEntry create-then-delete is unsafe for the filer sink: CreateEntry returns nil without creating on a transient chunk-copy error, so the paired delete could remove the only valid destination copy; a directory rename also deleted the old subtree before descendants were recreated, and left old chunks behind. Add an optional EntryMover sink capability and implement it on the filer sink via AtomicRenameEntry — one atomic, metadata-only move that relocates a whole subtree in a single transaction. Renames prefer it; sinks without a native move keep create-then-delete. When the old path is already gone (a descendant the parent rename moved, or one never replicated) MoveEntry creates the new path instead, re-checking existence with a lookup so a rolled-back move that left the old entry intact is retried rather than mistaken for gone. * docs(filer.sync): note entryMissing's gRPC not-found string fallback is deliberate	2026-06-09 12:54:28 -07:00
Chris LuandGitHub	7b07d8177a	fix(filer.sync): scope filesystem key sanitization to the local sink (#9894) ... * fix(filer.sync): scope filesystem key sanitization to the local sink destKey ran every sink key through escapeKey, whose Windows build strips colons. Colons are illegal in NTFS filenames so the local sink needs that, but s3/filer/azure/gcs/b2 accept them as ordinary key bytes — stripping them silently diverged the destination key (a source a:b replicated as ab). Move the sanitization into the local sink behind a Windows build tag, applied at every entry point so the previously-unescaped in-place-update paths stay consistent. Non-local sinks now keep the raw key; non-Windows builds are unchanged; a leading drive-letter colon is preserved. * test(filer.sync): cover incremental destKey and localsink update/delete sanitization Lock the colon-preserving behavior for the incremental destKey branch, and extend the Windows local-sink test to assert UpdateEntry and DeleteEntry also sanitize the key, not just CreateEntry.	2026-06-09 10:18:49 -07:00
Chris Lu	ed470dccb1	mini: grow volumes one at a time ... Mini auto-sizes a few large volume slots, but the master pre-grows 7 volumes per new collection. Under a filer group each S3 bucket is its own collection, so the first buckets claimed every slot and later writes failed to assign a volume. Cap mini's volume_growth copy counts to 1.	2026-06-08 14:51:40 -07:00
4 people	1b5f1c1f3b	feat(filer.backup): -initialSnapshot re-seeds a reinitialized destination (#9828) ... * feat(filer.backup): add -resetCheckpoint to force a fresh sync filer.backup resumes from a per-sink offset persisted in the source filer's KV. There was no first-class way to discard that checkpoint and re-run from the beginning short of guessing a large -timeAgo, which also skips -initialSnapshot. Add -resetCheckpoint: before reading the offset, write 0 for this sink so getOffset returns 0, isFreshSync stays true, and -initialSnapshot re-runs a full walk. Effective only when -timeAgo is 0. The flag is cleared after the first successful reset: runFilerBackup retries doFilerBackup forever on error, so leaving it set would re-zero the checkpoint on every retry and never make forward progress after a transient failure. Later retries resume from the persisted checkpoint instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(filer.backup): keep fresh-sync intent when offset read fails after reset After -resetCheckpoint writes offset 0, a transient getOffset read-back error flipped isFreshSync to false, which skipped the -initialSnapshot walk the reset explicitly requested. Track that the reset happened this iteration and, on a getOffset error, preserve isFreshSync=true in that case (the non-reset path keeps treating a read error as "not fresh" to avoid re-walking on transients). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(filer.backup): skip offset read-back on reset instead of tracking a flag Replace the didReset bool by branching: on -resetCheckpoint, clear the offset and start fresh without reading it back (we just wrote 0, so the state is known); otherwise read the offset as before. This drops the redundant getOffset RPC after a reset and removes the read-back error case entirely, so no separate flag is needed to preserve isFreshSync. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * filer.backup: -initialSnapshot re-seeds on every start; drop -resetCheckpoint -initialSnapshot now walks the live tree whenever -timeAgo is 0, seeds the destination, and overwrites the saved checkpoint, rather than running only on a fresh sync. That re-seeds a reinitialized destination on its own, so the separate -resetCheckpoint flag is gone. The walk runs once per process: the in-memory flag is cleared after the watermark is persisted, so the retry loop resumes from the persisted checkpoint instead of re-walking on every transient error. A process restart re-walks, so remove the flag once the backup is caught up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-06-07 23:35:53 -07:00
Chris LuandGitHub	89cbb1c558	admin: default -dataDir to "." so maintenance task state persists across restarts (#9856) ... admin: default -dataDir to "." so maintenance task state persists Previously -dataDir defaulted to empty, so the admin ran maintenance in memory only: task state was never saved and maintenance tasks (notably EC balance/rebuild) were re-issued every scan cycle without converging, churning EC shards (moves landed shards without their .ecx index, leaving EC volumes unloadable/missing shards). Default -dataDir to "." (the process working directory, which under the standard systemd unit is the admin's data dir) so state persists out of the box.	2026-06-07 20:45:03 -07:00
Chris LuandGitHub	755af4adf4	s3: actually bind outbound connections when -ip.bind is set (#9849) ... * s3: set outbound bind IP before the first filer dial Standalone weed s3 dialed the filer for GetFilerConfiguration before SetOutboundLocalIP ran, so that gRPC conn was created with the stock dialer and no source address. gRPC caches conns by address and reuses the original dialer on reconnect, so the s3->filer connection kept leaving from the OS-chosen source for the life of the process even after the bind IP was set a moment later. * grpc: install the outbound-bind dialer unconditionally The dialer was installed only when OutboundLocalAddr was already set at GrpcDial time, baking the source-address decision into the cached conn, so a conn dialed before the bind IP was configured never bound. Install the context dialer always and decide per dial: bind through OutboundDialContext once a source is set, otherwise fall back to the stock net.Dialer so default deployments keep gRPC's dial timeout and keepalive behavior. The bind now applies on the next reconnect regardless of ordering, matching the HTTP transport's unconditional DialContext.	2026-06-07 10:20:58 -07:00
Chris LuandGitHub	be7f417a03	ip.bind: bind outbound connections to the configured address (#9834) ... * ip.bind: bind outbound connections to the configured address -ip.bind only governed listeners; outbound gRPC and HTTP connections let the OS pick the source IP, which may not even be able to reach the target. Mirror the bind address into a process-global source address and apply it to outbound TCP dials: the gRPC context dialer, the per-client HTTP transports, and the default transport. Loopback targets and unix sockets keep the OS-chosen source so same-host traffic still works. * ip.bind: first-write-wins source IP, skip on address-family mismatch Make SetOutboundLocalIP first-write-wins so a `weed server` component's own bind setting (run in its goroutine) can't clobber the process-wide source address the top-level -ip.bind already established for the other components. Skip source binding when the target is a literal IP of a different family than the bind address, since forcing a mismatched source fails the dial.	2026-06-05 12:44:21 -07:00
Chris LuandGitHub	ab7be7867d	security: hot-reload JWT signing keys on SIGHUP (#9826) ... * security: reload JWT signing keys on SIGHUP Signing keys were read once in the server constructors and never refreshed. After a key rotation (Secret update, divergent reads) the in-memory key stayed stale and every request kept failing "wrong jwt" until the affected process was restarted. Add Guard.UpdateSigningKeys and call it from the master, volume and filer reload paths and the s3 reload hook, next to the existing whitelist refresh. Make the global chunk-read JWT cache reloadable via an atomic swap, and register the master's Reload with grace.OnReload -- it was never wired, so the master ignored SIGHUP entirely. Mirror the same refresh in the Rust volume server's SIGHUP handler. * security: swap signing keys behind an atomic pointer Addresses review feedback on the in-place key swap: SigningKey is a []byte, so reassigning the Guard fields while a request handler reads them is a data race that can tear the multi-word slice header and read out of bounds. Hold the four signing-key fields in an immutable signingConfig snapshot behind atomic.Pointer; UpdateSigningKeys swaps the whole pointer, so a reader sees either the old keys or the new ones. Reads go through new SigningKey/ExpiresAfterSec/ReadSigningKey/ReadExpiresAfterSec accessors. The Rust guard is already safe: every read and the SIGHUP write go through the shared RwLock<Guard>. * security: fold whitelist + auth state into the atomic snapshot Review follow-up. UpdateSigningKeys still wrote isWriteActive while the request path read it (and the whitelist maps) unsynchronized, so a SIGHUP under load could expose an inconsistent mix of activation bits and whitelist contents. Move all hot-reloadable Guard state -- keys, expirations, whitelist, and the activation flags -- into a single immutable guardState swapped behind one atomic.Pointer. The Update* methods take a small mutex to serialize the read-modify-write; readers stay lock-free. The concurrency test now also rotates the whitelist and probes IsWhiteListed under -race. Also read each signing key once per branch in the volume/filer JWT auth checks, so a reload landing mid-check can't take the allow-fast-path after auth was enabled or verify against a different key than the branch saw.	2026-06-04 22:26:08 -07:00
3 people	6e8002f065	fix: handle meta backup offset errors safely (#9818) ... * fix: log meta backup offset errors * fix: log meta backup offset errors * fix: exit on meta backup offset errors Exit with a non-zero status when the initial metadata backup offset cannot be persisted. Classify offset-read failures during streaming so the backup process exits instead of retrying forever, allowing supervisors to restart and bootstrap from a missing checkpoint. * meta backup: read offset in the loop, drop offset error type Reading the saved offset inside the retry loop makes an offset read failure a clean exit and a stream error a retry, without a typed error to tell them apart. streamMetadataBackup now takes the start time. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-06-04 10:53:23 -07:00
4 people	ce6a51468a	sftpd: support SSH user certificates signed by a trusted CA (#9815) ... * sftpd: support SSH user certificates signed by a trusted CA Adds a new "certificate" auth method to weed sftp. When enabled, the server loads trusted CA public keys from -trustedUserCAKeysFile (OpenSSH authorized_keys format, one or more keys) and accepts only ssh.Certificate blobs of type UserCert on the public-key channel. Validation uses ssh.CertChecker: CA signature, ValidAfter/ValidBefore, non-empty ValidPrincipals and SSH login user must appear in ValidPrincipals. The authenticated user must exist in the user store; home dir and permissions resolve as before. Behaviour mirrors MinIO's --sftp=trusted-user-ca-key and OpenSSH's TrustedUserCAKeys: when certificate auth is active, plain (non-cert) public keys are rejected even if "publickey" is also listed. Default authMethods remain "password,publickey", so existing deployments are unaffected. * Update weed/sftpd/auth/certificate.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * sftpd: address review feedback on certificate auth - Pre-marshal trusted CA public keys in IsUserAuthority instead of re-marshaling on every authentication attempt (gemini-code-assist). - Differentiate user-not-found from underlying store errors via errors.As(*user.UserNotFoundError) so backend/read failures are no longer reported as bad credentials (coderabbitai). - Fix the corresponding sanity check in the missing-file test to use errors.As instead of errors.Is (UserNotFoundError has no Is method, so the previous check never matched) (coderabbitai). * sftpd: register trustedUserCAKeysFile flag in filer and server commands The new field on SftpOptions is dereferenced unconditionally in resolvePaths(), but only the standalone `weed sftp` command was wiring its flag. `weed filer` and `weed server` both embed an SftpOptions value and call resolvePaths() on it, so they hit a nil pointer dereference at startup. Register `-sftp.trustedUserCAKeysFile` in both commands and update the -sftp.authMethods help text to mention the new "certificate" method. Fixes the SFTP Integration Tests CI failure on this PR. * helm: expose SFTP certificate auth in the SeaweedFS chart Adds Helm-chart support for the new SSH user-certificate auth method: - values.yaml (sftp:) gains `trustedUserCAKeys` (inline OpenSSH authorized_keys-format CA public keys) and `existingCAKeysSecret` (reference an externally managed Secret). Same pair added under allInOne.sftp with a null default that falls back to the top-level sftp.* setting. - New template templates/sftp/sftp-ca-secret.yaml renders a chart-managed Secret <release>-sftp-ca-secret with `ca_user.pub`, but only when SFTP is enabled, "certificate" is in authMethods, inline keys are provided, and no existingCAKeysSecret is set. - templates/sftp/sftp-deployment.yaml and the all-in-one deployment template add `-trustedUserCAKeysFile=/etc/sw/sftp_ca/ca_user.pub` to the weed sftp command, mount the CA secret at /etc/sw/sftp_ca and add the corresponding volume. All cert-auth bits are guarded by `contains "certificate" authMethods` so existing users see no change. - authMethods help text updated to mention "certificate". Verified end-to-end on a local k3d cluster: cert login succeeds, plain-pubkey login is rejected with "public key without certificate not allowed". * helm: fail render when SFTP certificate auth lacks CA keys When certificate is in authMethods but neither trustedUserCAKeys nor existingCAKeysSecret is set, the deployment mounted a secret that the chart never renders, leaving the pod stuck on a missing volume. Fail at template time with a clear message instead. * sftpd: fix stale auth-method list in SFTPServiceOptions comment keyboard-interactive was never implemented; certificate is the new supported method. Match the CLI help text. * sftpd: test Manager wiring of certificate vs public-key channel Cover the channel takeover at the Manager level: certificate auth displaces plain public-key auth when both are enabled, public-key auth stays put otherwise, and enabling certificate without a CA file errors. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-06-03 22:32:47 -07:00
4 people	e3e02d3364	[CheckDisk]: implement disk health detection (#9560) ... * [CheckDisk][GRPC]: implement MVP for disk health detection, added timeout for new grpc connections * fix(volume): build disk health check on every platform setDiskStatus only existed behind the statfs build tag, so disk.go failed to compile on windows, openbsd, solaris, netbsd and plan9. Move the timeout wrapper and failure tracking into the shared disk.go and have each platform's fillInDiskStatus return an error, so every platform gets the same protection from a stuck filesystem. Also restore the uint64(fs.Bavail) cast: Bavail is int64 on freebsd, so the unguarded multiply broke the freebsd build. * fix(volume): keep one outstanding statfs probe per disk A stuck statfs used to leave isChecking cleared by the timeout path, so the next check spawned another goroutine while the previous one was still blocked in the syscall, leaking one goroutine per minute on a hung disk. Clear the flag only when statfs returns and treat an overlapping check as a failure, so a hung filesystem keeps a single outstanding probe and still gets reported. * fix(volume): assume disk available until the first health check isDiskAvailable defaulted to false, and CollectHeartbeat skips locations that are not available. A freshly started volume server would therefore omit every volume from its first heartbeats until the async CheckDiskSpace ran, so the master could briefly treat all of them as missing. * fix(volume): label the disk error metric by data directory The new gauge tagged the series with IdxDirectory while every neighbouring resource gauge uses Directory, so the error series would not line up with them in dashboards. Also log the underlying error instead of a generic message. * test(volume): cover disk health success and repeated-failure paths * fix(volume): make a healthy disk the zero-value default Track the disk as isDiskUnavailable instead of isDiskAvailable so the safe state is the zero value, matching isDiskSpaceLow. CollectHeartbeat only skips a location once a check has actively marked it unavailable, so any DiskLocation built without running CheckDiskSpace (tests, future call sites) still reports its volumes instead of silently dropping them. * feat(disk): detect degraded disks using IO latency probes * feat(stats): introduce configurable disk I/O health probe with EWMA-based latency detection * feat(disk): replace EWMA with sliding window algorithm for disk health detection and added user-friendly options * feat(disk): improve disk health probing and recovery * feat(volume): configure disk health checks via volume.toml * fix(volume): Remove disk IO probe CLI options --------- Co-authored-by: ptukha <ptukha@tochka.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-06-02 09:02:05 -07:00
Chris LuandGitHub	2386fa550a	grpc: don't tear down the shared master connection on a caller's own timeout (#9775) ... A Canceled/DeadlineExceeded from the caller's per-request context was treated like a dead channel: it closed the shared cached ClientConn and cancelled every other in-flight RPC on it with "the client connection is closing". Under a burst of concurrent chunk assigns (e.g. a large S3 multipart upload) one slow assign hitting its 10s attempt timeout could poison the connection for all the rest, cascading into a flood of 500s. Thread the caller's context into shouldInvalidateConnection and only invalidate on Canceled/DeadlineExceeded while that context is still live, which isolates the genuine stale-channel signal (a peer restart behind a k8s Service VIP). To carry the context, add a ctx parameter to the existing WithGrpcClient, WithMasterClient, and WithMasterServerClient; the master assign and volume-lookup paths pass their per-attempt context and every other caller passes context.Background().	2026-06-01 15:11:02 -07:00
Chris LuandGitHub	80dd3b2621	EC bitrot follow-ups: protect destination sidecar on optional copy; cap sidecar block_size (#9763) ... * fix(ec_bitrot): cap sidecar block_size in ValidateBitrotManifest A sidecar loaded from disk (or supplied via a backfill/peer RPC) could carry a huge power-of-two block_size that passed validation, then force a multi-GiB scratch-buffer allocation in scrub/verify. Add a shared MaxBitrotBlockSize (64 MiB) constant, enforce it as an upper bound in isPow2MultipleOf1MiB, and derive the volume flag cap from the same constant so they cannot drift. * fix(ec_bitrot): don't destroy a valid destination sidecar on an optional copy writeToFile opened the destination with O_TRUNC before knowing whether the source had the file, so an optional copy (ignoreSourceFileNotFound) from a source that lacks the .ecsum truncated and then removed a valid pre-existing destination sidecar. Stage the optional copy into a temp sibling and commit it with an atomic rename only when the source actually delivered the file; a missing source is now a no-op. Mandatory copies keep their in-place behavior.	2026-05-31 23:42:33 -07:00
Chris LuandGitHub	9658f309d2	EC bitrot detection: per-shard checksum sidecars (#9761) ... * ec: add EC bitrot checksum protobuf EcBitrotProtection/EcShardChecksums/ChecksumAlgorithm sidecar messages, copy_ecsum_file and unsafe_ignore_sidecar fields, and a CHECKSUM scrub mode. * ec: bitrot checksum sidecar format, validation, and per-volume load Per-shard CRC32C block checksums in an optional <base>.ecsum sidecar with a self-integrity header; validation, rolling builder, backfill primitive, and EcVolume load on mount + removal on destroy. * ec: capture per-shard checksums at encode; verify-and-exclude on rebuild WriteEcFilesWithContext returns the protection computed inline during encoding. generateMissingEcFiles verifies present inputs against the sidecar, excludes corrupt ones, regenerates in place, and re-verifies; fail-closed unless unsafe_ignore_sidecar, removing all generated outputs on failure. * ec: read-only checksum scrub with Reed-Solomon arbiter ChecksumScrub verifies each local shard against the sidecar and reconstructs flagged shards from the clean shards so stale-sidecar false positives are not reported. Wired to the gRPC CHECKSUM mode and ec.scrub -mode checksum. * ec: server-side bitrot sidecar write, copy, cleanup, and opportunistic backfill Write .ecsum at fresh encode; propagate it with copy_ecsum_file (tolerant); remove it on full delete and decode; rebuild honors unsafe_ignore_sidecar and opportunistically backfills a sidecar when all shards are reachable. * ec: volume server bitrot config flags -ec.bitrotChecksum (default on) and -ec.bitrotBlockSizeMB (default 16). * fix(ec_bitrot): bound -ec.bitrotBlockSizeMB before the int64 multiply Validate the MiB value is in [1, 1024] before multiplying by 1 MiB, so a huge flag value cannot overflow int64 and slip past the power-of-two check, and a block size cannot collapse a sidecar to a few oversized blocks. * fix(ec_bitrot): distribute the .ecsum sidecar from the worker encode path The worker EC encode wrote the generation-0 sidecar locally but never added it to shardFiles, so DistributeEcShards never shipped it and the distributed holders came up unprotected. Append it to shardFiles and map the ecsum shard type to its extension in the sender so it travels with the shards. * fix(ec_bitrot): remove orphaned sidecars when the generation is gone Gate sidecar removal on existingShardCount==0 alone rather than also requiring a stray .ecx. A sidecar whose shards have all been deleted is orphaned and must be removed even when no .ecx remains, or it leaks. .ecx/.ecj/.vif removal stays gated on hasEcxFile as before. * fix(ec_bitrot): do not fold checksum blocks scanned into TotalFiles ChecksumScrub's first return is blocks scanned, not files. Discard it so the scrub response's TotalFiles (a needle/file count) is not inflated by the block count for CHECKSUM mode. * test(ec_bitrot): clean up generated .ecsum sidecars in removeGeneratedFiles * fix(ec_bitrot): reject an oversized sidecar payload before the uint32 cast The header stores payload_len as a uint32; bound the payload before the conversion so a pathological manifest cannot truncate the length field and corrupt the sidecar. A real manifest is a few KB, so this never trips. * fix(ec_bitrot): cap -ec.bitrotBlockSizeMB at 64 MiB The block size becomes the per-shard scratch buffer the scrub/backfill path allocates, so an over-large value (e.g. 1 GiB) is a memory hazard per concurrent scrub worker. Lower the upper bound from 1024 to 64 MiB. * fix(ec_bitrot): add -ecUnsafeIgnoreSidecar to weed tool fix -ecx The -ecx recovery path reconstructs missing shards via RebuildEcFilesWithContext, which fails closed on a malformed/stale .ecsum. Without an override flag an operator could not complete the rebuild without manually deleting the sidecar. Expose -ecUnsafeIgnoreSidecar (default false) and thread it through. * fix(ec_bitrot): bound sidecar payload with a direct int constant; drop readFull Guard len(payload) against a plain int constant (1 GiB) before the allocation instead of a uint64 MaxUint32 compare, so the allocation-size value is provably bounded (clears the CodeQL overflow alert) and the math import is no longer needed. Inline os.File.ReadAt with io.EOF handling in verifyShardFileBlocks and remove the now-redundant readFull helper (os.File.ReadAt fills the slice or errors). * test(ec_bitrot): use slices.Contains instead of a hand-rolled containsU32 * refactor(ec): fold the EcFiles WithContext variants into the base functions RebuildEcFiles now takes the *ECContext directly (nil => derive from .vif as before) and WriteEcFiles takes it too (nil => default), removing the parallel RebuildEcFilesWithContext / WriteEcFilesWithContext names. Callers that had an explicit context drop the WithContext suffix; the default-context callers pass nil. No behavior change. * refactor(ec): pass BackgroundECContext instead of nil to Write/RebuildEcFiles Add a non-nil BackgroundECContext placeholder (analogous to context.Background()) and have callers with no specific layout pass it instead of a nil *ECContext. WriteEcFiles resolves a zero/background context to the default ratio and RebuildEcFiles resolves it from the .vif, so behavior is unchanged. * fix(ec_bitrot): make BackgroundECContext a func; RebuildEcFiles fails closed on bad .vif - BackgroundECContext is now a function returning a fresh *ECContext, so callers cannot mutate a shared singleton or race on it (and it mirrors context.Background, which is also a function). - RebuildEcFiles now propagates the MaybeLoadVolumeInfo error: a present-but- unreadable .vif fails closed instead of silently rebuilding with the default ratio (which would corrupt a custom-ratio volume). Pass an explicit ctx to override.	2026-05-31 18:52:44 -07:00
Chris LuandGitHub	05c6500453	volume: fix maxVolumeCount dead zone that stalled writes on auto-sized disks (#9755) ... * volume: don't drop the last writable slot on auto-sized disks MaybeAdjustVolumeMax subtracted 1 from the per-disk slot count, so a disk with room for exactly one volume (free between 1x and 2x the size limit) reported 0 slots. The master then never grew a writable volume and every assign drained its retry budget, so writes failed with context deadline exceeded. Count the full volumes that actually fit, floored at one for an auto-sized disk that has free space. * mini: show disk and volume capacity in the startup banner Print free space, volume size, total volume count and free volume count under the data directory line, so a volume size limit that outstrips the disk is visible at startup instead of surfacing later as failed writes.	2026-05-30 23:45:17 -07:00
Chris Lu	5834c834e3	Refine enterprise edition feature blurb in version output and docs	2026-05-30 09:29:06 -07:00
Chris LuandGitHub	c9623007a2	fix(filer.sync): keep sync_offset fresh through filtered-event markers (#9733) ... On a read-only watched path the idle heartbeat keeps sync_offset fresh, but a busy source filer still emits a MaxUnsyncedEvents marker after many filtered events. The marker has a non-nil but empty EventNotification, so the client routed it to the event path, where it advanced no real watermark yet drove offsetFunc to republish the stale processed watermark — regressing the gauge between heartbeats and spiking the derived lag every time a filtered-event burst landed. Route the empty marker through OnIdleHeartbeat like the idle heartbeat so its fresh timestamp keeps the gauge current; it still advances the in-stream resume cursor.	2026-05-28 23:29:59 -07:00
Chris LuandGitHub	2f0643e5b1	fix(volume): stop flipping volumes read-only on a non-append-ordered .idx (#9726) ... * fix(volume): verify the .dat-tail needle in the integrity check CheckVolumeDataIntegrity checked the last entry by file position in the .idx and, for a live needle, flipped the volume read-only when fileSize > fileTailOffset. That entry is the .dat tail only when the .idx is in append order; a key-sorted .idx (weed fix and other rebuilds listed entries by key) puts the highest-key needle last, whose tail sits mid-file, so healthy volumes went read-only on every load and re-running weed fix only reproduced the sorted index. Locate the needle at the maximum offset — the one physically last in the .dat — and verify the .dat ends exactly at it, regardless of .idx ordering. The append-ordered common case stays O(1) (the last entry's on-disk end matches the .dat size); only a key-sorted index pays a single linear scan. Deletion tombstones at the tail are now verified too, instead of skipping the file-size check. * fix(command): weed fix rebuilds the .idx in .dat offset order SaveToIdx wrote entries via AscendingVisit — sorted by key, the .sdx/.ecx shape — so the rebuilt .idx put the highest-key needle last instead of the .dat-tail needle, and dropped tombstones whose live needle was gone. Collect the live and deleted entries, sort by .dat offset, and write them in append order so the .idx stays a faithful log whose last entry is the real .dat tail.	2026-05-28 18:04:31 -07:00
Chris LuandGitHub	dfd05d14cb	refactor(filer): remove the inode->path index and the NFS gateway (#9724) ... * fix(filer): derive inodes by hash instead of a snowflake sequencer Compute the same inode the FUSE mount would: non-hard-linked entries hash path + crtime, hard links hash their shared HardLinkId so every link resolves to one inode. Removes the snowflake inodeSequencer and the SEAWEEDFS_FILER_SNOWFLAKE_ID knob; inodes are now deterministic across filers. * chore: remove the experimental NFS gateway The NFS frontend ('weed nfs') was the only consumer of the inode->path index. Remove the weed/server/nfs package, the command and its registration, the integration test harness, and the CI workflow; go mod tidy drops the willscott/go-nfs and go-nfs-client dependencies. * refactor(filer): drop the inode->path index With the NFS gateway gone, nothing reads it. A regular file's inode is a pure hash of its path and a hard link's is a hash of its shared HardLinkId -- both derivable on demand -- so the secondary KV index and its write/remove hooks are dead. Removes filer_inode_index.go and the recordInodeIndex hooks from the store wrapper.	2026-05-28 15:00:18 -07:00
Chris LuandGitHub	3481f13f54	mount: route POSIX advisory locks to the owner filer under -dlm (#9669) ... With -dlm, GetLk/SetLk/SetLkw and the flush/release cleanup paths go to the inode's owner filer via the PosixLock RPC instead of the local table, so flock/fcntl are honored across mounts. Advisory locking rides the same switch as whole-file write coordination — and is therefore off under writeback cache, which implies single-writer. Keys are the inode identity (HardLinkId else path); SetLkw is client-side polling with the FUSE cancel channel (no server wait queue); a per-mount session id namespaces owners; a local hint avoids a release RPC on every close. Background unlock/release RPCs are bounded so a stuck filer can't hang close().	2026-05-24 23:56:37 -07:00
Chris LuandGitHub	25beb7ec48	admin: expose Prometheus metrics (#9652) ... * admin: add -metricsPort flag to expose Prometheus metrics The admin command had no metrics endpoint, so passing -metricsPort (as the operator does for spec.admin.metricsPort) crashed the process with "flag provided but not defined". Wire up -metricsPort/-metricsIp and start the shared Prometheus metrics server, matching filer, master, and volume. * admin: emit maintenance task and worker fleet metrics Add Prometheus metrics for the admin server's distinctive work: the maintenance task queue and the worker fleet that executes it. Task lifecycle: maintenance_tasks_by_status / _by_type gauges (snapshot of the queue), maintenance_tasks_completed_total{type,outcome} counter and maintenance_task_duration_seconds{type} histogram (recorded when a task reaches a terminal state), and last/next scan timestamp gauges. Worker fleet: workers_connected and worker_slots{used,max} gauges, plus worker_events_total{event} counting register/unregister/stale removals. Gauges are snapshotted by a background goroutine on the admin server; counters and the histogram are recorded at their event sites. * admin: read worker slot totals under lock, clear next-scan gauge when idle GetWorkers returns live worker pointers; summing CurrentLoad/MaxConcurrent outside the queue lock races with task assignment and completion. Add GetWorkerSlotTotals to aggregate under the lock. Also reset maintenance_next_scan_timestamp_seconds to 0 when the scanner is not running, so it can't retain a stale value after a stop.	2026-05-24 14:09:02 -07:00
Chris LuandGitHub	303c2be38d	feat(fix): rebuild lost EC index (.ecx) and .vif from local shards (#9596) ... weed fix -ecx reconstructs the .dat from the local data shards, scans the needles, and writes a fresh ascending-sorted .ecx containing only live entries — the same on-disk index WriteSortedFileFromIdx emits at encode time. When the .vif is also missing it is regenerated from the inferred EC ratio (flags > .vif > shard-count inference / 10+4) and the .dat size recovered from the scan. When some data shards are missing but at least dataShards shards survive, the missing shards are first reconstructed from the survivors via Reed-Solomon, so a partial shard set is repaired too. Also makes erasure_coding.WriteDatFile de-stripe using len(shardFileNames) instead of the DataShardsCount constant, so the caller's actual data-shard count is honored (behavior-preserving for the default 10, and fixing the existing caller that already passes ECContext.DataShards). This recovers an EC volume whose sealed index was lost from every node while enough shards survive, a state neither ec.rebuild nor ec.decode can repair because both require an existing .ecx. Flags: -ecx, -ecDataShards, -ecParityShards. Run with the volume server stopped.	2026-05-21 00:41:27 -07:00
Chris LuandGitHub	5af7d12f04	fix(filer.sync): keep sync_offset fresh while the source is read-only (#9589) ... * fix(filer.sync): keep sync_offset fresh while the source is read-only sync_offset holds the timestamp of the last replicated source event, so monitoring derives lag from now-sync_offset. A read-only source emits no metadata events, so the gauge froze at the last write and the derived lag grew without bound, making thresholds unusable. The source filer now sends an idle heartbeat carrying its current time while a subscriber is caught up to the buffer head. filer.sync uses it to advance the gauge, so now-sync_offset reflects real lag. Heartbeats are opt-in (client_supports_idle_heartbeat), are never written to the metadata log, and do not move the resume checkpoint, so a restart still resumes from the last real event. * fix(filer.sync): gate idle heartbeat on the read cursor, not SinceNs In metadata-chunks mode persisted entries replay as log file refs and never reach eachLogEntryFn, so lastSeenTsNs stays put and a caught-up subscriber with an old SinceNs would never get a heartbeat. Use the read cursor (lastReadTime), which advances in that mode too, max'd with lastSeenTsNs so the in-memory backlog-then-idle case still works while the cursor returned to the caller has not yet updated.	2026-05-20 11:26:37 -07:00
3 people	9914e6af30	chore(weed/command): prune unused functions (#9573) ... * chore(weed/command): prune unused functions * drop now-unused closed field and renderLocked guard --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-05-19 17:45:50 -07:00
Chris LuandGitHub	3d872a1416	fix(filer): load -s3.config static identities into the filer's CredentialManager (#9537) ... When weed filer started its embedded S3 gateway with -s3 -s3.config, only the S3 server loaded the s3.json static identities — the filer's own CredentialManager stayed empty, so the IAM gRPC service backing the admin UI and weed shell returned only dynamic users. Mirror the wiring weed server already does and hand the same config path to the filer.	2026-05-18 13:41:30 -07:00
Chris LuandGitHub	6b94701213	mini: quieter startup with a docker-compose-style progress board (#9524) ... * mini: quieter startup with a docker-compose-style progress board Replaces noisy startup/shutdown logs with a single in-place progress table on a TTY (or one line per state change off-TTY). Each component renders as `pending -> starting -> ready` during startup and `stopping -> stopped` during shutdown, with elapsed time on transition. Also folds in a few cleanups uncovered while making this readable: - route the admin.go startup prints through glog so quietMiniLogs() filters them under mini but standalone weed admin still shows them - generate a dev SSE-S3 KEK + passphrase on first run via WEED_S3_SSE_KEK and WEED_S3_SSE_KEK_PASSPHRASE env vars (viper.Set has a nested-key conflict between s3.sse.kek and s3.sse.kek.passphrase); persisted under the data folder so restarts reuse the same key - demote worker/master gRPC Recv 'context canceled' to V(1); those are the normal shutdown signal, not Errors/Warnings - drop the 'Optimized Settings' block and the 'credentials loaded from environment variables' message from the welcome banner - only show the credentials setup hints when no S3 identities exist (new s3api.HasAnyIdentity accessor backed by an atomic.Bool) - use S3_BUCKET in the credentials hint so it pairs with AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY - reorder running-services list to master / volume / filer / webdav / s3 / iceberg / admin * mini: refuse in-memory-only SSE-S3 dev keys; surface admin serve errors loadOrCreateMiniHexSecret returns "" when os.WriteFile fails, so SSE-S3 won't encrypt data under a KEK that the next restart can't reproduce (which would orphan whatever was written this run). The caller already treats "" as "skip setting WEED_S3_SSE_* env vars", so SSE-S3 and IAM just stay disabled for this run. startAdminServer's serve goroutine used to only log ListenAndServe failures, so a bind error left the caller blocked on ctx.Done() with no listener. Forward the error through a buffered channel and select on it alongside ctx.Done(). * ci(s3-proxy-signature): match weed mini's new progress-board ready line The readiness probe grepped for "S3 (gateway|service).*(started|ready)", which matched weed mini's old "S3 service is ready at ..." line. Mini now emits " S3 ready (Xs)" from its progress board, so the old pattern misses and the test timed out at the 30-second wait. Widen the alternation to also accept "S3\s+ready". The curl HEAD fallback already covers any remaining cases.	2026-05-17 19:13:09 -07:00
Chris LuandGitHub	62821964dd	filer/iam-grpc: make admin Bearer auth opt-in (fixes #9509) (#9514) ... PR #9442 made the filer refuse to register the IAM gRPC service unless jwt.filer_signing.key was set in security.toml, which broke the admin UI Users/Groups/Policies pages for every deployment that ships without a security.toml — weed mini, plain Helm, vanilla weed filer. The Users tab returns Unimplemented and the page is unusable. Issues #9504, #9505 and #9509 all trace to this gap. The rest of the filer's gRPC surface is unauthenticated by default; treat IAM the same way. The service now always registers, and the auth gate is a no-op when no signing key is configured. When the key is set, every RPC still requires an admin-signed Bearer token, matching the post-#9442 behaviour. Operators who expose the filer gRPC port beyond a trusted network should set the key on both filer and admin. The admin client (IamGrpcStore.withIamClient) already skips attaching the authorization metadata when its key is empty, so no changes there.	2026-05-15 13:15:20 -07:00