mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-16 12:16:36 +00:00
3ff6ce3c696b0a5b1dfa46411926c3dc7c8feecf
1529
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
c8cc56be91 | iceberg: route unprefixed requests to the first table bucket (#10675) | ||
|
Chris LuandGitHub
|
9d11278d95 |
filer: add filer.meta.scan to audit one directory's change history (#10645)
* filer: drain pending log chunk refs when the metadata stream ends In metadata chunks mode the server sends log file refs in responses of their own, and the client can only read them once it knows the run of refs is over. That was inferred solely from the arrival of a normal event, so refs still pending when the stream ended were dropped: the subscription returned no events and no error. A follower never noticed, because it runs forever and a live event always arrives to close the run. A bounded subscription — StopTsNs set, range already in the past — can receive nothing but refs and then EOF, and silently reports that nothing happened. For anything auditing a path that is the worst possible answer, since an empty result is indistinguishable from a quiet period. Drain on EOF as well as at the transition point. * filer: add filer.meta.scan to audit one directory's change history Reconstructing what happened to a path means replaying the metadata log, and filer.meta.tail is built for watching rather than auditing: it follows forever unless given a stop, prints multi-line JSON, and takes ranges only as durations before now, so an incident timestamp has to be converted by hand. Its -pattern also cannot find a versioned object. A versioned key is stored as <key>.versions/v_<id>, so the events carry the names "<key>.versions" and "v_<id>" and a pattern of the object's own name matches neither — the search comes back empty while the object is being written continuously. filer.meta.scan prints one line per change, stops at the end of the range, accepts absolute -since/-until with an explicit -tz, and reports versioned writes against the object key with the version id alongside, so -name matches the key a client would ask for. Delete markers are labelled as such rather than appearing as zero-length writes, and pointer flips on the .versions container are distinguished from writes of object data. * filer.meta.scan: read persisted log chunks from the volume servers Reading a range through the filer makes it decode every log entry in that range and filter each one, so the cost lands on the filer and does not shrink when the prefix is narrow — only the bytes on the wire do. On a cluster whose metadata log is dense that is the expensive part of a scan, and it is charged to the process least able to spare it. Enable metadata chunks mode: the filer hands out log chunk ids and the scan reads them from the volume servers itself. ReadLogFileRefs re-applies the same path filter client-side, so the output is unchanged — verified identical to the filer-read path over the same range, including after a restart drops the in-memory buffer and the data must come off disk. Direct read needs a route to the volume servers that the filer does not, so a failure before anything has been printed retries through the filer; retrying after partial output would duplicate lines. -directRead=false forces it. * filer.meta.scan: confirm an empty direct-read result through the filer An audit that returns nothing is read as "nothing happened here", so it is the one answer that must not be produced by a bug. Direct read has more ways to come back empty than the filer path does — it needs a route to the volume servers, and it depends on the ref-drain contract holding. When direct read yields no changes, re-run through the filer before reporting it, and warn if the two disagree. Re-running is safe only because nothing was printed; after partial output a replay would duplicate lines instead, so that case reports the error rather than retrying. |
||
|
Chris LuandGitHub
|
cab666fca1 |
filer: configurable TUS max upload size and session expiry (#10638)
* make TUS max upload size and session expiry configurable * default TUS session expiry to 24h |
||
|
Chris LuandGitHub
|
dd73fee077 |
mount: read oversized directories through instead of caching them (#10631)
* mount: read oversized directories through instead of caching them Visiting a directory pulls every child from the filer into the local LevelDB before the first listing returns. For a directory of a few million entries that is minutes of streaming, gigabytes of local store, and gigabytes of decoded entries in flight -- paid by a mount that may only walk the directory once. A build that crosses -cacheDirMaxEntries (default ten thousand) now stops, cleans up, and marks the directory read-through: listings stream from the filer with pagination, the way update-hot directories already do, and lookups in it consult the filer per entry as any uncached directory does. The refusal is remembered, so the next visit fails fast instead of streaming to the limit again, and an oversized ancestor is stepped over when caching its subdirectories rather than wedging every listing beneath it. The direct path keeps the same pagination state on the handle, so a walk that crosses the limit mid-flight carries on from where the cached walk reached. * mount: an ancestor found oversized must not fail its descendants Visiting a directory builds its whole uncached ancestor chain in one group, so the first discovery that an ancestor is oversized cancelled the group and surfaced as the listed directory's own refusal: the descendant build was aborted and the caller marked the descendant read-through, leaving a perfectly cacheable directory streaming from the filer until its inode was forgotten. The earlier test missed this by pre-marking the ancestor, which exercises only the fast path. The refusal of any directory other than the one being listed is now kept out of the group's result; it is already remembered for the next visit. |
||
|
Chris LuandGitHub
|
b8cba2982c |
mount: tell windows about changes made elsewhere (#10553)
* mount: tell windows about changes made elsewhere Nothing invalidates a Windows client's cache from this side, so a file created or removed by another mount, the S3 gateway or the filer API stayed invisible in Explorer until the user refreshed by hand. The mount already receives those events; they just had nowhere to go. WFS gains a listener for every applied metadata event, and on Windows that turns into the WinFsp notification for the path. A rename reports both ends, since the destination's own event may never arrive when it falls outside this mount. * mount: report a removed directory as a directory Entry is nil once a path is vacated, so asking it whether the thing that went away was a directory always answered no and every removal was reported as a file. Windows watches the two through different filters, so a folder removed elsewhere never refreshed. The invalidation now carries what used to be there, which the event already knew and simply was not passing on. * mount: report a rename destination once The event stream already carries a second invalidation describing the new path, so reporting RenamedTo here sent the destination twice — and always as a create, so a moved directory arrived as a create followed by a mkdir. |
||
|
Chris LuandGitHub
|
e377149d39 |
mount: support mounting on Windows through WinFsp (#10536)
* mount: add the WinFsp filesystem adapter WinFsp speaks a path-based FUSE dialect; weed/mount implements the inode-based raw protocol the Linux kernel uses. This translates between them so Windows runs the same filesystem code as everywhere else rather than a second implementation: paths resolve to inodes one Lookup at a time, and the raw operations run unchanged underneath. Errno translation is spelled out rather than passed through. Go numbers Windows errnos as offsets from APPLICATION_ERROR, so the raw value would mean something unrelated by the time WinFsp read it. Hard links return ENOSYS since WinFsp has none, and byte-range locks stay with its kernel driver rather than the mount's lock table. Not reachable from the mount command yet. * mount: build the winfsp errno table with explicit precedence Platforms alias errnos differently: freebsd has no ENODATA and linux makes ENOATTR the same value as it. A map literal with colliding constant keys does not compile, so build the table and let the first entry win, keeping the general codes their own meaning. * mount: wire the winfsp adapter into the mount command RunMount was one function doing filer setup, mount-point preparation and serving. The setup is the same everywhere, so it moves to mount_common.go and each platform keeps only what differs. Windows differs mostly in the mount point: WinFsp wants a drive letter or a path that does not exist yet, so none of the unix preparation applies, and a bad one is worth rejecting up front because WinFsp reports failure as a bare false. Adds -windows.caseInsensitive for software that expects Windows naming rules. * ci: mount on windows and exercise it Builds weed.exe, installs WinFsp, starts a cluster, mounts S: and runs a test suite against it: round trips at several sizes, offset writes, rename, delete, nested directories, concurrent writers, and a directory wide enough to stand in for the case that prompted this. Nothing else here can run the Windows mount, so without this the adapter is only known to compile. * ci: build the windows mount without cgo The runner has MinGW, so cgo is on by default and cgofuse compiles its cgo variant, which needs WinFsp's headers. The nocgo variant loads the DLL at run time and is what the released weed.exe uses. * mount: make the winfsp path splitting portable and test it resolve and resolveParent had the splitting inline in a windows-tagged file, so the cases that matter most there — both separators, empty and dot components, the root having no parent to create in — could not be tested on any runner that builds this. * test: check the windows mount persists across a remount Reading a file back through the same live mount proves nothing about durability; the answer can come from the mount's own caches. Write the fixtures, confirm the filer serves them with the mount out of the path, then re-read after a teardown and remount. * test: cover the windows mount operations that had none Truncate, append, chtimes and the hard-link refusal were implemented but never exercised, and the errno table was only unit-tested for mapping, never end to end. Adds names that have to survive the UTF-16 boundary, rename over an existing target and across directories, and concurrent handles on one file rather than one file each. * ci: dial the filer over ipv4 and run the persistence phases localhost resolves to ::1 first on windows and the cluster binds ipv4 only, so the mount's grpc dial was refused while the http readiness probe passed by falling back to ipv4. * ci: pin the cluster to loopback and probe ports by connecting weed mini advertises the runner's LAN address and binds filer grpc there, so the mount's dial to 127.0.0.1:18888 was refused while http answered. The readiness probe also passed with nothing on 18888: Test-NetConnection reported success for a port that then refused a connection, so it now opens a socket instead. * ci: report listening ports before mounting The readiness probe connects to the filer grpc port and the mount is then refused on it, which cannot both be true; print the actual state. * ci: run the cluster, mount and tests in one step The runner tears down a step's process tree when its shell exits, so the cluster started in an earlier step was already gone: the readiness probe passed against a live filer, the step ended, and the mount then found nothing listening. A diagnostic step reported no weed.exe at all. Everything that needs those processes alive now shares a step. * mount: key windows file io on the handle, not the path Read and Write walked the path on every call to fill in a NodeId the raw filesystem never reads: both look the file up by handle. Under eight writers creating files in one directory the walk transiently missed and the write failed with ENOENT before reaching the filesystem at all. Same for flush, fsync and the release calls. O_EXCL now fails on an existing name instead of taking it over, and Symlink is refused: the entry is easy to create but WinFsp only follows it once the reparse point is wired up, so it read back as an empty file. * mount: translate cgofuse open flags for windows cgofuse reports MSVC's numbering and the raw filesystem tests Go's, so only the access mode and O_TRUNC lined up: O_EXCL arrived as O_APPEND and O_CREAT as nothing at all. Also report which handle a failed write was using, to tell a handle that was never issued from one released while still in use. * mount: report which step of a windows create failed A concurrent create fails with ENOENT and the path walk, the parent lookup and the create itself are indistinguishable from the caller. * ci: send weed logs to stderr on windows glog writes to its own files by default, so the mount's own error output never reached the redirected log. Its flags are global and have to come before the subcommand. * mount: resolve known paths from the inode table on windows Every create walked the parent chain with a filer lookup per component. With eight writers creating files in one directory that is hundreds of concurrent lookups of the same parent, and lookupEntry reports an authoritative ENOENT when the directory is cached, the entry is not in the cache and the inode table has no record — a window a concurrent refresh can open for a directory that plainly exists. A path the mount already tracks now resolves straight out of that table. * test: sync the windows persistence fixtures before closing The mount is killed rather than unmounted, so anything still queued for flush is legitimately lost and the test was measuring crash durability while calling it persistence. A 9MB file lost four chunks that way. * mount: keep the lookup refresh on the target path Resolving a tracked path straight from the inode table skipped Lookup, which is also what refreshes the entry: a truncate then read back the pre-truncate size. Only the parent chain takes the shortcut now, which is where the concurrent creates were racing anyway. * mount: log every windows resolve failure Open suppressed ENOENT and Getattr logged nothing, which hid the two callbacks that can report a missing file during a create. * mount: drop dot entries from windows directory listings readdir reports "." and ".." for the kernel, but Windows enumerates a directory without them and displays whatever it is handed, so a folder of 200 files listed 202. Go's ReadDir filters them, which is why only the PowerShell walk caught it. * mount: flush queued writes when windows mount is interrupted The signal handler exits the process the moment its hooks return, so the WaitForAsyncFlush after Serve never ran on ctrl-c and queued writes were dropped. * mount: let windows mount over an empty directory WinFsp turns a directory mount point into a reparse point, which NTFS allows on an empty directory and refuses on a populated one. The check rejected every existing directory, so the ordinary habit of creating the mount point first failed with a message saying it should not exist. CI now mounts over a pre-created directory and writes through it. * ci: run the windows mount check on any pull request It is the only thing that exercises the Windows mount, so restricting it to pull requests based on master skipped it for stacked ones. Replaces the branch name that was pushed to trigger it. * mount: do not log a missing windows entry as an error Windows probes for entries that do not exist as a matter of course, so ENOENT from getattr and open is an answer rather than a fault and would have filled the log. * mount: take the fast path for parent chains in every windows resolve Narrowing it to resolveParent left Getattr and Open re-walking the parent with a filer lookup per component, and those are what Windows calls before a create: eight writers in one directory still raced a meta cache refresh there. Only the final component needs the Lookup refresh. The pass that suggested otherwise came from a run five times slower than the failing ones, where the race had no room to appear. * mount: drop the windows path resolution shortcut Resolving from the inode table skipped the Lookup that refreshes an entry, and a truncate then read back its old size. Applying it only to the parent chain kept truncate correct but left concurrent creates failing, and applying it to the final component too inverted that. The two cannot both be satisfied this way, so this returns to looking up every component and leaves the concurrent create failure open. * mount: fall back to the open handle when a deferred entry is evicted A create that defers the filer write leaves the entry only in the local cache. Creating many files at once pushes the directory past the hot threshold and evicts it, taking that placeholder with it, so a lookup went to the filer, found nothing, and reported a file that plainly exists as missing. The handle still holding the unflushed entry is authoritative for it. Caught by concurrent creates over a Windows mount, which resolves a path on every call rather than relying on a kernel dentry cache. * mount: let cgofuse resolve to the version the module graph requires rclone already depends on cgofuse at a newer commit than the v1.6.0 pin, so readonly builds refused the go.mod until it matched what MVS picks. The interface and flag values the adapter uses are unchanged there. * mount: wait for a pending async flush before looking up on the filer Open, unlink and rename already wait, but a plain lookup went straight to the filer and read pre-close metadata: truncate a file, close it, and a path probe during the flush window reported the old size. The kernel attr cache hides this on linux; a front end that resolves paths on every operation hit it directly. * mount: reject a umask wider than the file mode it becomes ParseUint allowed 64 bits and the result is narrowed to os.FileMode, which is 32, so an out-of-range umask truncated silently instead of being reported as unparseable. * mount: address review findings on the windows mount WaitForAsyncFlush closed its channel unconditionally and shutdown reaches it from both the interrupt hook and the path that resumes after serving, so a ctrl-c could panic on a second close. The deferred-entry fallback read an open handle's entry without its lock, which is what the other two readers of that field take so FromPbEntry does not walk the chunk slice mid-append. The async-flush wait also sat ahead of the meta cache, making every stat of a recently closed file queue behind uploads; it belongs just before the filer is consulted. Windows entries were persisted as uid 0: the raw filesystem stores InHeader's owner and the adapter left it zero. They now carry the identity the mount was started with. The errno table used Linux numbering while cgofuse decodes MSVC's, so ENAMETOOLONG arrived as EDEADLK and five others were likewise wrong; a windows test pins each value to cgofuse's own constant. Also: break the filer handshake loop on success rather than always running ten rounds, accept a drive letter written S:\\, report a missing WinFsp instead of panicking, keep commas out of the volume label, and drop -windows.caseInsensitive, which told WinFsp the mount folds case while lookups stayed exact. * mount: return windows lookup references so the inode table stays bounded Every operation that hands back an EntryOut grants a reference the Linux kernel returns with FORGET. WinFsp has no FORGET, so the adapter took one per path component per call, plus one per child of every readdirplus, and never gave any back: inodeToPath grew for the life of the mount. Walking the 200k-file directory this exists for stranded 200k references. The adapter now plays the part the kernel plays. Each resolution releases what it took, and an open handle keeps the reference for its inode until Release, counted because the raw filesystem reuses one handle for repeated opens. Holding it is not optional: completeAsyncFlush skips the metadata flush when the saved path no longer maps to the inode, so releasing early would lose a close's metadata. Also stops persisting the display owner. -o uid=-1 makes WinFsp report the calling user whatever we say, but the value handed to the raw filesystem is written to the filer, and 4294967295 is what every other client would read. -windows.uid and -windows.gid set what is recorded. * mount: fix windows behaviours the reference implementations guard against WinFsp has no ro option — it discards the flag and leaves the volume writable — so -readOnly accepted writes and deletes. The refusal now happens in the operations themselves. Windows sends times around its own 1601 epoch, which arrive as a large negative second count; casting them through stored a year-1601 timestamp that every other client then read. Those are now left alone. rclone carries the same guard. Chown returned ENOSYS, and WinFsp passes a chown failure straight out of SetSecurity, so Explorer's Security tab and icacls failed for edits that were not about ownership. It now accepts and discards. Only create and mkdir presented a caller; the rest sent uid 0, which hasAccess treats as root, so deletes and renames skipped the permission check that creates got. Every operation presents the same identity now. A drive letter written S:\ reached WinFsp unnormalised, which recognises a drive only as exactly two characters and then failed as a directory path. A test also pins the open flag translation, since swapping O_EXCL and O_TRUNC would turn 'fail if it exists' into 'truncate it'. * mount: answer windows getattr and truncate from the open handle WinFsp keeps the path a handle was opened with and never updates it when the file is renamed, so resolving the path again fails on a handle that is still perfectly valid — the ordinary write-temp-then-rename save pattern. The handle already knows its inode, which also removes a full path walk from two operations WinFsp calls constantly. Readlink on the root now refuses. WinFsp probes there to decide whether the volume has symlinks and enables them unless it fails, and with them on it resolves a path a component at a time, each one reaching us as its own walk — all for a feature Symlink already refuses. * mount: require the windows mount directory not to exist WinFsp creates the directory itself with FILE_CREATE and removes it when the filesystem goes away, so an existing one — empty or not — fails with "mount point in use". Allowing an empty directory was wrong, and the CI check that appeared to prove otherwise was the vacuous one: listing a plain directory succeeds whether or not anything is mounted on it, so the step passed while the mount had failed and the writes went to local disk. That check now waits for the reparse point, which is what caught this. * mount: apply review comments on the windows mount -windows.uid and -windows.gid reached the adapter but not the filesystem parameters, which is what carries the owner written to the filer, so the flags changed nothing. Readdir re-resolved the path while Getattr and Truncate answer from the handle; a directory renamed during an enumeration then failed on the stale path WinFsp still holds. Utimens now honours UTIME_OMIT instead of writing whatever came with it. * mount: tag the unix-only lock tests away from windows The production lock files were tagged when the package was made to build on windows, but the tests that exercise them were not, so anything that compiles tests for windows still failed on syscall.F_WRLCK. * ci: vet the mount tests for each target too Only compiling the non-test build let an untagged test keep a per-OS syscall constant without anything noticing. |
||
|
Chris LuandGitHub
|
46ceb253b0 |
telemetry: report anonymous cluster stats by default (#10488)
The reports are what tell us which versions and cluster sizes are actually in use, and almost nobody flips the flag on, so the numbers we have are close to useless. Default it on for master, server and mini, and say in the flag help and the startup log how to turn it off. Nothing new is collected: still an in-memory cluster id that changes on restart, version, os, server counts, volume count and disk bytes, sent once a day by the leader master only. |
||
|
Chris LuandGitHub
|
167c114dae |
ci: fix FUSE mounts against the new runner image (#10484)
* ci: restore the setuid bit on a shadowed fusermount3 Newer ubuntu-22.04 runner images carry a source-built fusermount3 in /usr/local/bin that shadows the distro one in PATH and is not setuid root. go-fuse looks the helper up through PATH, so every unprivileged mount fails with "mount failed: Operation not permitted". * test: fail a fuse test as soon as its mount process dies A mount that cannot mount at all exits within a second, but the harness still waited out the 30s readiness timeout and then reported "mount point not ready within timeout", leaving the real cause buried in the log tail. Watch the child processes and report their exit instead. * mount: report a failed mount without a goroutine dump A mount failure is an environment problem - no /dev/fuse, fusermount not setuid, stale mount point - and the all-goroutine stack dump Fatalf adds buries the one line that says so. |
||
|
Chris LuandGitHub
|
4149346bb7 |
s3: register the advertised ip with the master (#10482)
* s3: register the advertised ip with the master The cluster address came from the bind ip, falling back to the auto-detected interface, so -ip never reached the S3 registration. weed mini -ip=localhost binds the wildcard and ended up registering whatever interface happened to sort first -- on a host with VPN interfaces, an address that stops routing once the tunnel drops. IAM changes are pushed to registered S3 servers over gRPC, so every mutation then blocked the full 10s propagation deadline before logging a failure, and cluster.ps and the admin UI listed a node nothing could reach. Identities still arrived through the /etc/iam metadata subscription, so this cost latency and visibility, not credentials. Add an advertise ip to the gateway option, preferring it over the bind address, and wire the parent -ip through server, filer and mini. * s3: treat any unspecified bind address as a wildcard net.ParseIP + IsUnspecified covers ::, [::] and the expanded IPv6 forms instead of only the 0.0.0.0 literal, so an IPv6 wildcard bind no longer registers an address peers cannot dial. Host names parse as nil and stay addresses in their own right. Apply the same guard to the advertised ip. |
||
|
Chris LuandGitHub
|
5536d88fbb |
azure: let the blob endpoint be configured (#10460)
* azure: let the blob endpoint be configured The service url was always derived as <account>.blob.core.windows.net, which leaves out Azure Government, Azure China, and private endpoints. Name the blob service url instead and those accounts become reachable. The url has to be https, since the account key or the bearer token would otherwise travel in the clear. * azure: reject an endpoint that carries no hostname A url like https://:443/ has a host of ":443", so the emptiness check on Host let it through and the request only failed once it reached Azure. The hostname is what has to be there. |
||
|
Chris LuandGitHub
|
fee3fcb55a |
mount: report data sizes to df with -df.logical (#10459)
df on a mount shows the space the cluster gives up to the data: every replica of a regular volume, every shard of an ec one. That is the honest answer for capacity planning, but it is not the question a user asks when they want to know how much of their data is stored. Add -df.logical. The master reports the logical sizes alongside the raw ones: one replica per regular volume, the data shards of each ec volume counted once. Free space is divided by the copies the requested replication makes, so used plus available stays the amount of data the mount can still write, and it comes off the cluster-wide usage rather than one collection's, since capacity is cluster-wide too. Statistics through a filer resolves an unset replication to the filer's default rather than the master's, matching where the writes it is sizing for actually land. The flag governs the quota check too, so a mount has one notion of how much it is using. A filer that predates the new fields sends zeros, and the mount keeps reporting the raw sizes. |
||
|
Chris LuandGitHub
|
3ae4e9c563 |
azure: authenticate with Entra ID instead of a storage account key (#10456)
* azure: authenticate the blob sink with Entra ID Shared account keys have to be distributed and rotated everywhere a sink runs. Leaving account_key empty now falls back to the identity chain, so a workload identity or managed identity carries the authorization instead. * azure: authenticate remote storage with Entra ID The remote storage client demanded an account key and refused to start without one. Fall back to the identity chain when it is absent, and let azure.client_id pin a user-assigned identity. * azure: reject a malformed storage account name The account name is interpolated into the service URL, so a name carrying a "/", "?" or "@" moves the authority elsewhere and an authenticated request follows it. Hold callers to Azure's own naming rule instead. * azure: keep a leftover environment key off the identity path A configured client id asks for Entra ID, but AZURE_STORAGE_ACCESS_KEY still filled in the account key behind it. An old mounted secret would go on authenticating until it rotated, and the failure then blamed the key. * azure: say what the identity path reads from the environment A pinned client id alone is not enough for workload identity: the tenant and the projected token come from the environment, and missing them only surfaces later, when a token is first requested. |
||
|
Chris LuandGitHub
|
83f754763e |
filer: make the redis connection settings configurable (#10441)
The sentinel stores hardcoded a 30s read timeout and a 1m retry backoff. After a sentinel failover every request that picked a pooled connection to the old master sat there for 30s before the connection was retired, and the pool timeout derived from it (read timeout + 1s) queued the rest behind them. The other redis stores took the go-redis defaults with no way to tune anything. Read the dial, timeout and pool knobs from each redis store section instead, keeping the go-redis default for every key left unset. |
||
|
Chris LuandGitHub
|
6824619c16 |
s3: chunk uploads at the filer's maxMB (#10439)
The S3 write path cut fixed 8MB chunks, so an object stored through S3 chunked differently from the same bytes stored through the filer, WebDAV or a mount, and -maxMB had no effect on it. Read maxMB from the filer configuration at startup and use it, falling back to 8MB when the filer reports none. |
||
|
Chris LuandGitHub
|
652273301e |
filer sync: do not advance the sync offset past a failed event (#10424)
* util: retry transient errors, not just the ones containing "transport" util.Retry only retried when the error string contained "transport", so a plain "read: connection reset by peer" from S3 got zero retries. Classify the error instead: net timeouts, connection resets, and the throttling and overload replies S3 and gRPC return are all worth another attempt, while a cancelled or expired context is not. * filer sync: hold the sync offset behind a failed event A sync job that returned an error was logged and forgotten, and the watermark advanced past it anyway. The offset is the durable resume point, so the event was never replayed: for filer.remote.sync that left the file present locally, absent on the remote, with no RemoteEntry and nothing to retry it. Pin the watermark at the oldest failed event. Later events keep flowing, but the persisted offset stays behind the failure, so a restart replays it. |
||
 TJDawson10andGitHub
|
b50116ccae |
fix(redis2/redis3): support separate sentinel auth credentials (#10412)
redis2_sentinel and redis3_sentinel stores only passed Username/Password into redis.FailoverOptions, which authenticates against the Redis master/replica servers. When Sentinel itself requires auth (requirepass set in sentinel.conf), go-redis had no credentials to send to it, causing a NOAUTH error before ever reaching the master. Add sentinel_username/sentinel_password config options that map to go-redis's SentinelUsername/SentinelPassword fields, distinct from the existing master auth credentials. |
||
|
Chris LuandGitHub
|
47b491b53c |
mount: version open file handles by filer log position (#10403)
* filer: stamp a log position on lookup and remote-cache responses Metadata events are logged after their store write and stamped with the filer clock. Reading that clock before serving an entry therefore gives a timestamp with a causal guarantee: every event at or below it is reflected in the returned entry. Clients caching filer state can use it as the entry's version to order the response against subscription events, including events committed before the call but delivered after it. * mount: version open file handles by filer log position A subscription event refreshing an open handle did a second lookup; a transient failure left the handle pinned to its old entry with no retry, since the subscription cursor had already advanced. The deeper problem is ordering: the handle is a cache written by three unordered channels — the async invalidation worker, local mutation acks, and open-time lookups — and overwriting cached state safely requires knowing which write is newer. The filer log timestamp is that order, and it now travels with every value instead of being derived out of band. Events carry it natively; lookup and remote-cache responses carry the log position stamped before the serving read; mutation acks carry it in their returned event; and the local store pairs each read with a version cursor advanced under the same lock as the store write. Each handle records the version its entry reflects, and one rule replaces the per-site reasoning: state at or below the handle's version is old news and must not be installed. The invalidation itself applies the event's own entry — no lookup, so no transient-failure window — except under a cached parent, where the store entry is the ordered merge of the event and anything applied since, and its version outranks the event's. An uncached parent receives no store writes, so a hit there would be a stale leftover masking the event. A vacated path (delete, rename away) keeps the last entry so unlinked-but-open reads still work. Directory builds version the completed directory at the listing snapshot and re-invalidate buffered events at that version, since their mid-build refresh ran against an incomplete store. The tests replay every race this replaces machinery for: rollback of a newer local flush (queued, cached, and read-through), stale leftovers under uncached parents, the build window including abort, handles opened after an event was queued, events landing mid-lookup, and undelivered events at remote-cache time across a filer failover. * filer: serialize the log position fence with mutations, stamp mutation acks The fence stamped before an unlocked entry read could precede state the read returned: a mutation writes storage first and assigns its event timestamp only at notify time, so a lookup racing that window handed the mount an entry newer than its fence, and the event's later delivery looked like fresh news — destroying dirty pages for a change the handle already had. The mutation handlers already hold an exclusive per-path lock across read, write, and notify; the lookup and remote-cache reads now take it shared around the stamp and the read, making the fence exact: everything at or below it is in the entry, nothing above it is. A no-change update returns success without an event, leaving the mount nothing to fence with even though the response confirms current state. Create and update acks now carry a log position stamped under the same lock, and the mount falls back to it whenever the ack has no event. Also regenerate the VT marshalers, which the earlier generation missed: without them a VT round-trip silently zeroed every log position. * java: sync filer.proto * mount: scope store versions to what they vouch for; atomic handle install The store's version cursor claimed too much. Advanced by local mutation acks and directory listing snapshots, it inflated the version of store reads for unrelated paths whose events the subscription still owed, and those events were then fenced out permanently. The cursor now tracks subscription progress only — events arrive in log order, so everything at or below it has been delivered for every path — and a completed listing records its snapshot as a per-directory floor instead of a global claim. Local acks never touch it: they version their own handle directly. Buffered build events advance the cursor at delivery, since their store write may never happen (abort) while their invalidation is already queued; their read-through directory pairs no store read with it, and rename fragments are applied first. Concurrent first opens raced: a slower opener's older lookup could overwrite the newer entry a faster opener had installed, while the monotonic version kept the newer timestamp — an old entry fenced at a new version, immune to every correcting event. Entry and version are now installed as one decision under the handle map lock, and an install that does not outrank the handle's version is dropped. The remote-cache commit also escaped the fence: it wrote storage and notified without the path lock, so a lookup's shared-locked fence and read could land between the two and hand out the cached state under-versioned. The commit now re-reads and writes under the exclusive path lock, and backs off entirely when the entry changed during the download — the concurrent writer supersedes the cached content. * mount: floors gate store applies; installs respect handle users; renames join the fence A directory floor certifies the listing state as of its snapshot, but a delayed event at or below the floor was still applied to the store — rolling the content back to pre-snapshot state while the floor kept claiming the snapshot version, so the correcting events were fenced out of every future read. Events are now gated against the affected directory's floor, each half of a rename independently. Fences are lower bounds: a listing or lookup can include a mutation whose event has not been delivered yet, and that event later passes every gate carrying state the handle already holds. Such a re-delivery now advances the version without destroying dirty pages or reinstalling the entry — invalidating local writes over a no-op was the real damage in every remaining under-fence window, including the unlocked listing snapshot, which no per-path lock can serialize. The concurrent-open install moved from the map lock to the handle lock every reader, writer, and invalidation synchronizes on, and rejects what cannot improve the handle: dirty state (local writes would be lost), unversioned lookup responses (they cannot outrank anything, and two zero-version opens must not overwrite each other), and anything not strictly newer. New handles are still fully initialized before the map exposes them. Renames committed metadata and emitted events with no path lock, so a lookup could read the renamed state under a fence preceding its events. Both rename handlers now hold the source and destination locks, ordered by path, across commit and notification; descendants of a renamed directory are not individually locked and rely on the no-op re-delivery handling above. * mount: per-entry store versions replace the cursor and directory floors The store's aggregate versions — a global subscription cursor and per-directory listing floors — were versions at coarser granularity than the values they described, and every over-claiming bug in this series traced to that gap: an aggregate vouching for state its source never saw. Each store entry now carries the filer log position of the write that produced it — the event that applied it, or the listing snapshot that inserted it, recorded in the store's key-value space under the same lock as the entry write. The store becomes what the handle already is: a last-writer-wins register with one rule, install only what outranks the current claim. The cursor, the floors, their advancement rules, the pairing ordering constraint, and the floor gating all collapse into that rule. Applies are gated per entry, each half of a rename independently; an unversioned local write clears the claim its content no longer proves; version records lingering after a bulk folder wipe cannot fence a recreate, since a claim only blocks while its entry exists. Listing inserts are stamped at build completion, before the buffered replay so newer replayed events override the stamp. Filer side, the fence dance every versioned read must perform is now a single choke point, fencedFindEntry, so a future read RPC gets the lock-serialized stamp by construction rather than by convention. * mount: judge no-op re-deliveries against an immutable base, not the live entry The equal-state skip compared the incoming event to the live handle entry, but local writes mutate the live entry — size, timestamps, chunks — so a delayed event re-delivering the base the handle was opened with no longer matched, and the installer destroyed the dirty pages and rolled the entry back over nothing new. The handle now keeps an immutable snapshot of the filer state it last installed or acknowledged, refreshed at every install and mutation ack (flush acks snapshot the request entry before the id mapping mutates it), and the no-op judgment runs against that base: an event carrying the base brings nothing, whatever the live entry has diverged to since. * mount: tombstones for versioned deletes, absence floors, copy enrollment Four gaps in the per-entry version protocol, all the same shape: a versioned fact with nothing carrying its version. A deletion is a fact about a path with no entry left to hold it — clearing the record let a delayed older event resurrect the deleted path, permanently, since the deletion's own redelivery is dedup-suppressed. Versioned deletes now leave a tombstone record that fences without an entry; renames tombstone their source the same way. Plain records still only block while their entry exists, so records lingering after a bulk folder wipe cannot fence a recreate. A completed listing proves absences as well as presences: a name it omitted was deleted as of the snapshot, and a delayed create below the snapshot re-creates it. The snapshot is kept per directory strictly as an absence fence, consulted only when a path has neither an entry nor a version record — present entries carry their own versions and never touch it, which is what separates this from the over-claiming floor it replaces. A rebuild against a pre-upgrade filer returns no snapshot; stamping now clears the children's records in that case, so a reinserted entry cannot reactivate the stale claim its previous incarnation left behind and reject valid events below it. Server-side copies installed the copied entry without enrolling in the base protocol, so the copy's own event differed from the stale pre-copy base and destroyed writes made to the destination after the copy. The install now refreshes the base and takes its version from the fenced readback. * mount: deletion facts outlive the cache's knowledge of the entry A versioned delete of a path the store held no entry for recorded nothing, so a delayed older event recreated the path — permanently, with the deletion's redelivery dedup-suppressed. The tombstone is now written whenever a versioned event vacates a path: the deletion is a fact about the path, not about what this cache happened to hold. For an absent entry, the listing's absence floor now speaks whatever older record remains: a tombstone at one position does not exhaust what is known about the path when a newer snapshot has confirmed the name still absent, and an event between the two was slipping past both. A committed copy whose readback failed installed a synthesized base with local timestamps; the copy's real event legitimately differs from it, and was read as foreign state — destroying writes made to the destination after the copy. The handle now marks that its own event is en route and adopts that event's state as the base without touching the live entry or the dirty pages; the adoption is one-shot, so a genuinely foreign event still invalidates. * mount: authoritative acks cancel pending event adoption; tombstones scoped and pruned The copy-event adoption flag could outlive its purpose: a flush after the failed readback installs a newer base and advances the version, the copy's own event is then version gated without consuming the flag, and the next genuinely foreign event was silently adopted — base advanced, live entry and dirty pages untouched — leaving the mount to later overwrite that remote change. Every local acknowledgment now installs its base through one helper that also cancels any pending adoption: the ack supersedes the mutation the adoption was waiting for. Tombstones were written for every versioned delete under the mount and survived directory eviction by design, growing LevelDB with historical deletions on delete-heavy mounts. They are now scoped to directories whose cached state the fence actually protects — an uncached parent never serves from the store nor applies the resurrecting insert — and a completed listing prunes the direct-child tombstones its absence floor supersedes, leaving only those above the snapshot. The store gains a key-prefix visitor for the sweep. * mount: acked saves install their value; trailer snapshots; direct-child prune range A version must never advance without its value. saveEntry stamped any open handle with the acknowledgment's version, but a handle opened while the save was in flight holds the pre-mutation entry — stamping it fenced out the events carrying the state it lacked, permanently, with the local apply performing no invalidation and the redelivery deduplicated. The acknowledged entry is now installed together with its version, through the same guarded install the racing-open path uses: under the handle lock, only when it outranks the handle, never over dirty local writes. Empty listings return no in-band snapshot — a snapshot-only response would be read as an entry by older consumers — so directories that end empty gained no absence floor and their tombstones were never pruned. The filer now sends the snapshot in the stream trailer, which older clients ignore, and the client reads it when no in-band snapshot arrived. Empty directories get real floors, their tombstones prune, and their buffered replays gain the snapshot filter instead of the replay-all fallback. Version records now encode the parent directory and name separated by a NUL, making a directory's direct children one contiguous key range: the tombstone prune scans exactly them under the cache lock, instead of walking every descendant record — the whole store, for root. * mount: fix dirty-page loss, uid/gid base, download race, copy adopt, leak; dedup Correctness fixes from the versioned-invalidation review: - A foreign delete/rename-away of a file held open with unflushed local writes destroyed the dirty pages unconditionally. A process may keep writing to an unlinked-but-open file and those writes were already acknowledged; preserve the pages when the handle is dirty. - downloadRemoteEntry stored the handle's base with filer-side uid/gid while every candidate it is later compared against is in local form, so under a non-identity UidGidMapper an unchanged re-delivery looked foreign and force-destroyed dirty pages. Map the base to local. - downloadRemoteEntry wrote the entry/base/version triple under only the handle's shared lock, so two concurrent reads of the same remote-only file could tear it. Serialize the install with a dedicated mutex (invalidation is already excluded by the exclusive handle lock). - A committed server-side copy whose readback failed adopted the FIRST event past the version gate as its base; a foreign write delivered first was silently swallowed. Adopt only an event whose content matches the synthesized base — the copy's own event — and install any other normally. - The deferred-create path relied on AcquireFileHandle installing the passed entry on a pre-existing handle, which the version rework dropped. Restore that install in the compat wrapper; the versioned open path keeps its gated install. Growth and hot-path cost: - Per-entry version records and tombstones leaked when a directory was evicted or read-through without a rebuild. An uncached directory gates its own inserts, so its records fence nothing; clear a directory's child version records when it is wiped for eviction. - FindEntry paid for the version KvGet on every lookup/getattr cache hit and threw it away. FindEntry now reads only the entry; the hot lookupEntry cache-hit path skips the version entirely. Cleanups: - Extract ackVersionTsNs over the shared response interface, replacing the metadata-event-else-log-ts snippet copy-pasted at four ack sites. - Extract acquireRenamePathLocks, replacing the verbatim sorted two-path lock fence in both rename handlers. * mount: no resurrection on foreign delete, version no-event acks, gate downloads, tighten copy adopt Follow-ups to the review patches: - Preserving dirty pages on a foreign delete let the next flush pass the isDeleted guard and CreateEntry, resurrecting the remotely-unlinked name. Mark the handle deleted in the vacate branch: the open fd can still read its buffered writes, but a flush no longer recreates the file. - A no-event acknowledgment (log fence only) synthesized a metadata event with TsNs 0, so the cache stored the entry unversioned and an older subscriber event rolled it back. Stamp the synthesized event with the ack's log position at all four ack sites. - downloadRemoteEntry serialized its install but did not check the version, so an older response arriving last overwrote the entry/base while the monotonic version kept the newer value, fencing corrections out. Install only when the response is at least as new as the handle. - sameEntryContent compared only size and chunks, so a foreign chmod with unchanged content was adopted as the copy's own event. Compare everything except server-assigned timestamps, so a metadata-only foreign change installs instead. * mount: trim comments to the non-obvious why The versioning work accumulated multi-line comment blocks restating what the code says. Keep the constraint a reader cannot derive — why a fence is exact, why a version must not advance without its value, why an uncached parent's records fence nothing — and drop the rest. * mount: distinguish rename from delete, tighten the download and adopt gates - A rename emits a nil old-path invalidation just like an unlink, so the vacate branch marked the handle deleted and later writes through the already-open descriptor were skipped instead of persisted. Carry the delete/rename distinction on the invalidation and mark only an actual delete. - The remote-download install accepted an unversioned response regardless of the handle's version, so during a rolling upgrade a delayed response could install stale content under a newer version. Require the response to be at least as new, with one exception: a handle still lacking local chunks takes the content anyway — it cannot read without it — but does not claim the response's log position. - Copy-event adoption returned without installing, so a foreign touch arriving before the copy's own event lost its timestamps. Content is unchanged either way, so the dirty pages stay valid; a clean handle now takes the entry, while a dirty one keeps its diverged version. * mount: one directory floor instead of a record per child; agree on TTL Review feedback: - Build completion wrote one KV record per direct child inside the cache write lock, so a large directory stalled every other cache operation for O(children) store writes. The directory's listing snapshot already covers every child it saw; make that floor the version for any child without a record of its own, and a child earns a record only when a later event touches it. One map write per build replaces the per-child writes, with the same fencing. - The presence probe read the store directly and so counted a TTL-expired entry as present, judging the path by a record describing content that has logically vanished. It now applies the same expiry the read path does, and an expired path falls back to its directory floor. - Preserve ErrNotFound identity when the commit-time re-read finds the object deleted, so callers still surface a 404. - Assert the rename-away source fence timestamp in the invalidation test. Also record the tombstone ceiling: distinct deleted names in a cached directory accumulate until it is rebuilt or evicted, which prunes everything at or below the new snapshot. * mount: pin the fence's clock domain instead of letting skew decide A log-position fence is stamped by one filer's clock under that filer's in-process lock, so comparing it to an event another filer logged is comparing two unrelated clocks. The two error directions are not equally costly: applying an event the fence already covered is a re-apply the base-equality check absorbs, while skipping one it does not cover leaves the handle holding exactly the state the event was meant to correct, with the subscription cursor already past it — the unhealable staleness this whole PR exists to remove. So refuse to guess. Fences now carry the signature of the filer that stamped them, and a handle records it alongside the position. An event is only fenced out when the filer that logged it is the one that stamped the fence — the logging filer appends its own signature, so its presence identifies the clock domain. Events from any other filer are applied. Positions taken from events keep comparing as before; the subscription already delivers those in order. The invalidation callback takes a struct now: it carries the path, entry, position, delete/rename distinction, and signatures, and was about to need a fifth positional parameter. * mount: follow a foreign rename; key page invalidation on content, not equality - A rename's old-path invalidation now carries the destination, and the handle follows the file there: an open fd tracks the inode, and leaving it on the old path made its next flush recreate that name instead of updating the renamed file. - Dirty pages overlay content, so only a content change invalidates them. Keying that on exact equality meant any timestamp-only event destroyed them, which the copy-adoption marker existed to paper over — a foreign touch could consume the marker and leave the copy's own event to drop the post-copy writes. Comparing content instead makes the marker unnecessary, so it is gone: a metadata-only event keeps the overlay, and a dirty handle keeps its diverged entry unless foreign content supersedes it. - A remote download response that is merely older is now refused even when the handle still lacks chunks; only an unversioned one is taken (and claims no position), since an older response's content predates what the handle reflects. - A refused or unversioned download no longer publishes to the metadata cache, where a zero-position event would clear the entry's version and let an older subscriber event roll the cache back. * mount: page invalidation keys on content alone; unversioned writes claim no position - sameEntryContent compared everything but timestamps, so a foreign chmod, chown, or xattr change counted as a content change and destroyed the dirty-page overlay. It was strict only to serve the copy-adoption marker, which is gone; its one caller now asks the question it actually needs — did the bytes change — so metadata-only events leave the overlay alone. - A rename over an existing file destroys that file, but its open handle was left live and still pointed at the name the renamed source now occupies, so its flush could overwrite it. MovePath already reports the displaced inode; mark that handle deleted. - An acknowledgment was refused whenever its position was numerically lower, even when a different filer stamped the fence it lost to. Two known, differing signatures mean unrelated clocks, so the comparison no longer applies there; unknown signatures still compare as before. - A local write with no log position behind it now records that explicitly instead of deleting its version record. Absence means the directory listing covers the path, which is why the snapshot floor applies; local content the listing never saw must not inherit it, or the events that would correct it are fenced out. * mount: widen the existing lookup functions instead of forking WithVersion twins The versioning work grew a parallel function for every accessor that needed to return a log position — lookupEntryWithVersion beside lookupEntry, maybeLoadEntryWithVersion beside maybeLoadEntry, FindEntryWithVersion beside FindEntry, AcquireFileHandleWithVersion beside AcquireFileHandle, advanceEntryVersion beside advanceEntryVersionTsNs, plus a getPbEntryWithVersion wrapper and an InsertListedEntriesForTest hook. Two names for one operation is two places to keep in step, and the split let callers pick the one that happened to compile. Each pair is now the single original name carrying the position, with callers that do not want it discarding it. filer_pb.GetEntry returns the fence its response already carried rather than a mount-side wrapper re-issuing the lookup, and InsertEntry takes the position its content reflects rather than a test-only twin that inserted without one. The one behavioural knot the merge exposed: AcquireFileHandle had been installing the entry on a pre-existing handle only in its unversioned form, which conflated 'the caller is authoritative' with 'the lookup had no version'. Deferred create is the only caller that means the former, so it now installs explicitly and the map function just acquires. |
||
 
|
490379bff3 |
Add codespell support with configuration and typo fixes (#10393)
* Add GitHub Actions workflow for codespell on master * Add rudimentary codespell config * Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers like allLocations, publishErr, ReadInside, FlushInterval. Also skip templ-generated *_templ.go files, and whitelist a handful of short/domain-specific words (visibles, fo, te, ser, bject, unparseable, keep-alives, tread, anc, ue) that show up as false positives across the tree. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix ambiguous typos and protect false positives Fixes typos that codespell reports with multiple candidate suggestions (so `codespell -w` cannot auto-apply them), plus one inline pragma and one config entry to protect legitimate identifiers. Manual fixes (single correct answer chosen from context): - pattens -> patterns (5x) in filer/upload/shell flag help strings - finded -> found (2x) in tarantool storage.lua comment - spacify -> specify (2x) in helm chart values.yaml comment - wether -> whether in skiplist.go docstring - simpe -> simple in mq schema test case name False-positive protection: - Add `//codespell:ignore` next to `source GET's` (possessive of HTTP verb) in s3api_object_handlers_copy_stream.go - Whitelist `auther` in .codespellrc — it's a local variable meaning "authenticator" in weed/security/tls.go, not a typo of "author". Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Extend codespell ignore list: .git-meta path and thirdparty groupId Also skip `.git-meta` (scratch dir for commit messages that may contain typo words verbatim) and whitelist `thirdparty` — it appears as the literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms and cannot be renamed. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w Auto-applied fixes to the 44 remaining single-suggestion typos across docs, comments, log messages, tests, config, and one Java pom. === Do not change lines below === { "chain": [], "cmd": "uvx codespell -w", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^ * Revert breaking codespell fixes; whitelist unknwon and atleast Two of the auto-applied `codespell -w` fixes were false positives that would break the build/tests: - go.mod: `github.com/unknwon/goconfig` is a real Go module path — the upstream author's GitHub handle is literally `unknwon`. Renaming to `unknown` would fail dependency resolution. - test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}: `atleast` is a literal CLI mode value (a string constant compared and passed as a positional argument). Rewriting to `at least` splits it into two arguments and breaks the mode check. Reverted those files and whitelisted both words in .codespellrc so future runs won't re-suggest the same broken fixes. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
   
|
6c4eb95a3a |
fix(admin): implement ApplyPluginConfigFromToml to propagate settings (#10388)
* fix(admin): implement ApplyPluginConfigFromToml to propagate settings to plugin config store * Update weed/admin/dash/config_toml.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * admin: overlay admin.toml onto plugin configs through bootstrap defaults Creating a config from scratch at startup skipped the descriptor-defaults bootstrap, so a job type with only worker keys in admin.toml persisted Enabled=false and RetryLimit=0 and silently stopped running. Overlay existing configs at startup, and apply the same overlay in enrichConfigDefaults when the plugin bootstraps a fresh config from descriptor defaults. Also place collection_filter in the admin values where workers read it, map preferred_tags as a string list, and stamp UpdatedAt. * admin: trim the admin.toml help text and call-site comment * admin: clamp toml retry values to the int32 range * admin: fail startup when admin.toml cannot reach the plugin config The legacy overlay already aborts startup when declared settings cannot persist; continuing here would let workers bootstrap with stale values. * admin: fix the retry clamp test on 32-bit A 32-bit int cannot hold the oversized toml value, so viper returns 0 before the clamp runs. --------- Co-authored-by: baracudaz <baracudaz@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
5a54beac80 |
EC decode: read shards with the encode-time block layout (#10385)
* erasure_coding: WriteDatFile takes the encode-time dat size for the shard block layout * volume server: derive EC decode layout from the encode-time dat size, not the live extent * erasure_coding: test decode after tail deletions shrink the live extent below a large-block row * seaweed-volume: write_dat_file_from_shards takes the encode-time dat size for the shard block layout * seaweed-volume: derive EC decode layout from the encode-time dat size, not the live extent * seaweed-volume: test decode after tail deletions shrink the live extent below a large-block row * erasure_coding: reject decoding with no data shards * worker: record the encode-time dat size in the .vif * erasure_coding: fall back to the shard-derived layout only when the encode-time dat size is missing * erasure_coding: reject an ambiguous shard-derived block layout * seaweed-volume: fall back to the shard-derived layout only when the encode-time dat size is missing * seaweed-volume: reject an ambiguous shard-derived block layout |
||
|
Chris LuandGitHub
|
76ec1d8f0f |
s3: accept raw semicolons in query strings (#10305)
* s3: accept raw semicolons in query strings Go's url.ParseQuery drops any key=value pair containing a raw ';'. A presigned PUT that signs content-type carries X-Amz-SignedHeaders=content-type%3Bhost; when a client or proxy decodes the %3B, the parameter vanished and the upload failed with MissingFields, while AWS accepts the raw ';' as query data. Re-encode it before routing so the pair survives parsing and signature verification. * iam, iceberg: recover raw-semicolon query pairs on the other listeners The standalone IAM API verifies SigV4 with a canonical query recomputed from the parsed query, and Iceberg REST warehouse/parent values may legally contain ';'. Move the normalization middleware to util/http and attach it to both routers. |
||
|
c17aebeecf |
fix(filer.backup): prevent silent backup data loss on transient not-found (#10295)
* fix(filer.backup): stop silently dropping events on transient not-found Under a write burst, filer.backup could consume a metadata event (advancing the persisted offset) without replicating the file, with no error logged: 1. filersink CreateEntry/UpdateEntry swallowed replicateChunks errors (glog.Warningf + return nil), so the offset advanced past entries that were never written. 2. The manifest-chunk branch of replicateChunks resolved via LookupFileId with no retry, unlike the data-chunk branch — transient lookup races dropped exactly the large manifest-backed files while small inline-content siblings landed. 3. isIgnorable404 matched "LookupFileId" / "volume id ... not found", misclassifying those races as genuine source 404s at the backup layer. Fix: on a replicateChunks failure the filer sink now skips only when the live source has moved past the replayed version (deleted or strictly-newer mtime) — lossless, a later event carries the current content — and propagates otherwise so the event is retried. The manifest resolve retries transient errors like the data-chunk path. isIgnorable404 is narrowed to genuine 404s; non-filer sinks and the initial-snapshot walk, which relied on the broad match as their only lossless-skip valve, now make the same live-source decision (filersink.SourceSupersedes) instead of retrying forever on a permanently gone volume. Tests cover propagation of unconfirmed lookup failures, the narrowed 404 classification, and the supersession guards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filer.backup): derive supersession path with directory fallback eventSourceSuperseded built the source path from NewParentPath alone. Legacy metadata events (persisted by older filers) carry an empty NewParentPath, so the probe looked up "/<name>", read the miss as "source gone", and skipped a live file on a transient lookup error — the silent drop this change is meant to eliminate. Derive the path via MetadataEventTargetFullPath (the same directory fallback genProcessFunction uses) and cover both event shapes with TestEventSupersessionProbe_PathDerivation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filersink): retry manifest resolve only for transient errors, bounded when unverifiable The manifest-resolve retry stopped only when hasSourceNewerVersion proved the source moved past the replayed version, which wedged the sink in two cases: incremental sinks use dated target keys that cannot map back to a source path (supersession never provable), and permanent resolve errors (corrupt manifest data, bad file ids) fail forever while the source entry stays live. Gate the retry instead: keep retrying only transient errors (volume-lookup races, network interruptions), stop after a few attempts when supersession cannot be checked, and propagate everything else immediately so the configured metadata error policy applies (-disableErrorRetry included). Propagation is lossless: filer.backup's fallback decides with the event's real source key, and both filer.backup and filer.sync re-deliver the event (RetryForeverOnError) without advancing the offset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filer.backup): make error classifiers nil-safe isIgnorable404, isSourceLookupError, and isTransientResolveError called err.Error() without a nil guard. All current call sites pass a non-nil error, but the guard is free and matches isRetryableNetworkError. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
9d75048594 |
admin: respect filerGroup for cluster discovery (#10170)
* Respect filerGroup in admin discovery Admin discovery previously queried master cluster nodes with an empty filer group, so filers registered under a non-default group could not appear in the admin UI. Add an admin filerGroup flag and carry it through cluster-node discovery requests while preserving the empty default behavior. Constraint: SeaweedFS master ListClusterNodes filters by exact filer_group. Rejected: Discover all groups implicitly | no existing admin or shell behavior exposes cross-group discovery. Confidence: high Scope-risk: narrow Directive: Keep admin cluster discovery scoped to the configured filerGroup unless an explicit all-groups API is added. Tested: docker run --rm -v "$PWD:/src" -w /src golang:1.25 go test ./weed/admin/dash -run TestListClusterNodesRequest -count=1 Tested: docker run --rm -v "$PWD:/src" -w /src golang:1.25 go test ./weed/command -run '^$' -count=1 Not-tested: full repository test suite * mini: pass filer group to admin cluster discovery miniAdminOptions.filerGroup was never initialized, so startAdminServer dereferenced a nil *string. Share the filer.filerGroup flag pointer so the co-located admin queries the same group the filer registers under. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
ebeab4b6ec |
feat(filer.sync.verify): reclassify chunk-slice-order ETag diffs as CHUNK_REORDER (#10177)
* feat(filer.sync.verify): reclassify chunk-slice-order ETag diffs as CHUNK_REORDER filer.ETagChunks concatenates per-chunk MD5s in stored slice order without normalising by offset, so byte-identical content written by two different paths (S3 multipart part-completion order on the source vs filer.backup replication arrival order on the destination) yields different file ETags. filer.sync.verify reported these as ETAG_MISMATCH even though the files are equal. Add a second-pass check on every ETAG_MISMATCH: when both sides derive their ETag from chunks (no attr.Md5) and hold a manifest-free, non-overlapping chunk set that, once sorted by offset, matches element-wise on (offset, size, ETag), classify the file as CHUNK_REORDER. Such files are content-equal, so they are not counted as errors and do not affect the exit code; they are listed only at higher verbosity (weed -v=1), while the summary always shows their count. The check stays conservative: a stored attr.Md5 (order-independent content hash), a differing chunk count, an overlapping/duplicate offset (whose visible bytes are resolved by timestamp), or a manifest chunk all remain ETAG_MISMATCH. * filer.sync.verify: decline chunk-reorder fast path on empty per-chunk ETag An empty or undecodable per-chunk ETag is not a content fingerprint, so element-wise (offset, size, ETag) equality can't prove the bytes match. Treating "" == "" as content-equal could reclassify a genuine divergence as CHUNK_REORDER and drop it from the error count. Decline such chunk sets so they stay ETAG_MISMATCH. * filer.sync.verify: emit CHUNK_REORDER in JSON output regardless of -v The -v=1 gate belongs to the human text report only. Applying it before the jsonOutput branch dropped the per-file CHUNK_REORDER records from NDJSON while the summary still counted them, so a machine consumer saw a non-zero count with no records to reconcile it. Gate the text path only; JSON always emits. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
 
|
bdcc3154ed |
refactor: centralize genUploadUrl in UploadOption (#10164)
* refactor: centralize genUploadUrl in UploadOption Replace inline genFileUrlFn closures with operation.GenUploadUrl field: - Add GenUploadUrl func(host, fileId) string to UploadOption struct - Add GenUploadUrlProxy(filerAddress string) utility function - Remove genFileUrlFn parameter from UploadWithRetry signature - Update all callers: mount, gateway, mq, filer_copy, filer_sync This matches the weed mount -filerProxy pattern exactly, factorizing the URL generation logic across all consumers. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * docker release: run all platform jobs in one wave, cache rocksdb compile Drop max-parallel so the 13 per-platform builds run together instead of two waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late). Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is sha-independent, so it caches across releases and stops being the ~16-min long-pole that gates the merge fan-in. go-build variants stay mode=min. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * refactor: centralize genUploadUrl in UploadOption Replace inline genFileUrlFn closures with operation.GenUploadUrl field: - Add GenUploadUrl func(host, fileId) string to UploadOption struct - Add GenUploadUrlProxy(filerAddress string) utility function - Remove genFileUrlFn parameter from UploadWithRetry signature - Update all callers: mount, gateway, mq, filer_copy, filer_sync This matches the weed mount -filerProxy pattern exactly, factorizing the URL generation logic across all consumers. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * Remove accidental ROCmFPX submodule reference * gofmt chunk upload option block * Preserve broker cipher and re-read proxy filer per upload attempt Chunk uploads must keep the configured Cipher, and both the mount and broker current filer can change on failover, so build the proxy upload URL inside the closure instead of capturing the address once. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
7c3c5ed2a4 |
fix(filer.sync.verify): sort listings client-side before merge (#10117)
* fix(filer.sync.verify): sort listings client-side before merge The merge walks both filers' directory listings in lockstep and needs them in the same byte order. A filer before 4.32 with a locale SQL collation lists case-insensitively while a 4.32+ peer lists byte-ordered, so comparing two such clusters returns the same names in a different order and the merge desyncs into spurious MISSING / ONLY_IN_B. Buffer and sort each directory client-side so both sides agree on order regardless of filer version or store backend. Trades the streaming source's O(buffer) memory for O(directory) per side, fine for a one-shot verify CLI; both sides still load concurrently. Claude-Session: https://claude.ai/code/session_01BKsBdKYFNCEjeHLjJfumPF * fix(filer.sync.verify): surface listing errors before merging A listing that fails mid-stream leaves a partial, unsorted buffer. Now that both sides are fully buffered anyway, check each side's error right after the loads finish and before the merge, so partial entries can't emit spurious MISSING / ONLY_IN_B before the error aborts the run. Claude-Session: https://claude.ai/code/session_01BKsBdKYFNCEjeHLjJfumPF |
||
|
Chris LuandGitHub
|
5112da98a2 |
mount: skip redundant permission checks under default_permissions (#10089)
With default_permissions (the mount default) the kernel enforces unix permission bits from the getattr/lookup attributes before it ever calls Open, Create, or Mknod. The mount was re-checking permissions in AcquireHandle and createRegularFile anyway, which duplicated the kernel's work and kept the supplementary-group lookup on the per-file hot path. Gate only the mode-bit access check on default_permissions being off, so a non-root copy does no permission work on open/create. createRegularFile still loads the parent to validate it exists, since the create RPC skips the filer-side parent check. With default_permissions off the mount remains the sole enforcer, so the full check still runs. |
||
 
|
e1f89f85f2 |
fix(filer): apply -filer.disk default to metadata log assigns (#10080)
* fix(filer): apply -filer.disk default to metadata log assigns Metadata event log writes call operation.Assign directly and used only FilerConf path rule DiskType. When filer.conf rules were missing or unmatched, the master received an empty DiskType and grew volumes on the built-in hdd layout. Mirror resolveAssignStorageOption: wire FilerOption.DiskType into the Filer, fall back when the matched path rule has no disk type, and return the matched rule from resolveMetadataLogAssignDiskType to avoid duplicate MatchStorageRule lookups. Co-authored-by: Cursor <cursoragent@cursor.com> * mini: fall back to -volume.disk for filer default disk type weed server copies -volume.disk into the filer disk default when -filer.disk is unset; weed mini did not, so metadata-log assigns sent an empty disk type on clusters that only tag volumes (e.g. hot/warm). --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
 
|
091d953c34 |
fix(benchmark): close CPU profile file handle after profiling (#10048)
Co-authored-by: Contributor <contributor@example.com> |
||
|
Chris LuandGitHub
|
53342c9ba6 |
mini: resolve admin credentials from security.toml and env vars (#10021)
* mini: resolve admin credentials from security.toml and env vars weed mini started the admin UI without resolving admin.user/admin.password (and the read-only pair) from security.toml [admin] or WEED_ADMIN_* env vars, so the only way to protect the UI was the -admin.password flag. The standalone weed admin command applies these fallbacks in runAdmin via applyViperFallback; the mini path calls startAdminServer directly and skipped it, leaving authRequired false and the UI unauthenticated. * mini: load admin.toml maintenance settings The mini admin path runs ApplyMaintenanceConfigFromToml (via startAdminServer) against the global viper, but runMini never merged admin.toml, so file-based maintenance task settings ([maintenance.vacuum], .balance, .erasure_coding) were ignored under mini while the standalone weed admin honored them. Load it alongside master/volume config. * mini: support -admin.urlPrefix for the admin UI Expose the reverse-proxy subdirectory prefix that the standalone weed admin already supports, so the mini admin UI can run under e.g. /seaweedfs. The prefix is normalized the same way and passed through to startAdminServer. |
||
|
Chris Lu
|
443b5b1184 | Merge branch 'master' of https://github.com/seaweedfs/seaweedfs | ||
|
Chris Lu
|
0c343e76eb | admin: don't log normal 2xx/3xx HTTP requests (incl. 304 cache hits) | ||
|
Chris LuandGitHub
|
e411ff491d |
volume: remove ec.bitrotChecksum and ec.bitrotBlockSizeMB flags (#10000)
EC bitrot protection is now a fixed default: always on at 16 MiB block granularity. These volume-server flags exposed needless configurability; the package defaults in erasure_coding (BitrotProtectionEnabled, BitrotBlockSize) are retained and still drive sidecar generation. Drop the now-unused erasure_coding import. |
||
|
Chris LuandGitHub
|
5e8152b81c |
storage: register tier backends at the binary composition root (#9989)
The s3 and rclone tiered-storage backends were registered via blank imports in weed/storage (volume_tier.go and volume_info/volume_info.go). That forced every library consumer of weed/storage -- weed/shell, and through it external tools -- to link aws-sdk-go and, under the rclone build tag, the full rclone backend set and its cloud-storage SDKs, even though those consumers never tier volumes. Move the registrations into a new weed/storage/backend/all aggregator and blank-import it once from weed/command, the binary's composition root. The weed binary still registers both backends; weed/storage and its library consumers no longer pull the backend SDKs into their dependency graph. |
||
|
Chris LuandGitHub
|
9266aaa88e |
security.toml: document WEED_ env override for jwt signing keys (#9981)
security.toml: document WEED_ env override for the jwt signing keys These keys are HMAC secrets; spell out the env-var mapping so they can be injected from a secret store instead of living in the config file. |
||
 
|
3bf3d29058 |
fix(command): preserve fuse option after writers (#9972)
* fix(command): preserve fuse option after writers Problem: FUSE mount option parsing skipped the option immediately following concurrentWriters, so values such as concurrentReaders could be silently ignored. Root cause: runFuse incremented the options loop index inside the concurrentWriters case in addition to the loop increment. Co-authored-by: Codex <noreply@openai.com> * test: save and restore mountOptions pointers, not their values The fields are reassigned to fresh heap variables during runFuse, so dereferencing to back up/restore mutated throwaways instead of the flag-bound originals and could nil-panic on unset fields. --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
37962e2445 |
admin: configure maintenance tasks via admin.toml (#9926)
* admin: configure maintenance tasks via admin.toml Maintenance task settings could only be edited in the admin UI and live under <dataDir>/conf, so they silently reverted to defaults whenever the data directory was recreated. An optional admin.toml now declares vacuum, balance, and erasure coding settings; keys set there are written through to the persisted task configs at every startup, overriding UI edits, so the configuration stays declarative. Generate an example with "weed scaffold -config=admin". * vacuum: round min volume age up to whole hours MinVolumeAgeSeconds was truncated by integer division when converted to the hour-granular protobuf field, so a sub-hour setting silently became 0 and disabled the age guard. * admin: split and normalize preferred_tags from admin.toml A comma-separated string, as set via environment variable, came through viper as a single slice element. Split on commas and reuse util.NormalizeTagList, matching the plugin config path. * scaffold: clarify admin.toml wording |
||
|
Chris LuandGitHub
|
e56a1c4c05 |
admin: pre-gzip embedded static assets, add cache headers (#9918)
The admin UI served embedded static files uncompressed and without cache headers: embed.FS has zero mod times, so no Last-Modified, no ETag, no 304s -- every page load re-downloaded ~700KB of css/js in full, which gets painful over slow or tunneled links. Gzip the static tree at generation time (go generate ./weed/admin) and embed only the compressed mirror, shrinking the binary ~1.5MB. The handler hands the pre-compressed bytes to gzip-capable clients, decompresses for the rest, and sets Cache-Control, per-variant content-hash ETags and Vary so repeat loads revalidate with a 304. bootstrap.min.css goes 232KB -> 30KB on the wire. A drift test keeps static_gz/ in sync with static/. |
||
|
Chris LuandGitHub
|
2ac5aa72c7 |
add elastic8 filer store for Elasticsearch 8 (#9916)
* elastic: fix listing against a missing or empty directory index
The refresh 404 leaked into the named return, so the first listing of a
directory whose index does not exist yet returned an error instead of an
empty result. Sorting also fails on an index with no documents
("No mapping found for [_id] in order to sort on"); unmapped_type
keeps the resumed-listing path working there.
* add elastic8 filer store for Elasticsearch 8
Elasticsearch 8 disables _id fielddata by default, so the elastic7
store's directory listings fail with "Fielddata access on the _id
field is disallowed". elastic8 uses the same client and configuration
options, but also indexes the document id as an Id field and sorts
listings on Id.keyword.
|
||
|
Chris LuandGitHub
|
e12052ee6b |
fix(filer.sync): replicate a rename as an atomic move, not a no-op update (#9895)
* fix(filer.sync): replicate a rename as create-then-delete, not an in-place update A rename arrives as a single metadata event carrying both the old and new entry. The filer sink was routed to UpdateEntry, which looks up the old path but issues the update against the new parent without changing the name — and the filer UpdateEntry RPC cannot move an entry. So the rename was dropped: the old path lingered and the new path never appeared (same-dir renames rewrote the old name in place). Route a real move (the sink path changed) through CreateEntry(new) then DeleteEntry(old) in both the replicator and the filer.sync/backup driver, the way the other sinks already handle it; reach UpdateEntry only for true in-place updates. Create before delete so a crash between the two leaves the entry visible rather than lost. * fix(filer.sync): derive the rename delete key like the create key, guard the watched root The rename delete leg rebuilt the old key with a raw util.Join, bypassing the sink-side key normalization the create leg gets from buildKey — so a rename could create the new entry and then fail to delete the old one under a transformed key. Build the old key through buildKey too, and skip the delete when the moved entry is the watched root itself (where the old key would resolve to the target root and recursively delete the whole sink tree). * test(filer.sync): cover the in-place update delete-then-create fallback order The recording sinks always reported foundExisting, so the fallback that an in-place update takes when the entry is missing on the sink was never run. Make it configurable and assert the fallback deletes before it recreates the same key, in both the replicator and the filer.sync drivers. * feat(filer.sync): move filer-sink renames natively via AtomicRenameEntry create-then-delete is unsafe for the filer sink: CreateEntry returns nil without creating on a transient chunk-copy error, so the paired delete could remove the only valid destination copy; a directory rename also deleted the old subtree before descendants were recreated, and left old chunks behind. Add an optional EntryMover sink capability and implement it on the filer sink via AtomicRenameEntry — one atomic, metadata-only move that relocates a whole subtree in a single transaction. Renames prefer it; sinks without a native move keep create-then-delete. When the old path is already gone (a descendant the parent rename moved, or one never replicated) MoveEntry creates the new path instead, re-checking existence with a lookup so a rolled-back move that left the old entry intact is retried rather than mistaken for gone. * docs(filer.sync): note entryMissing's gRPC not-found string fallback is deliberate |
||
|
Chris LuandGitHub
|
7b07d8177a |
fix(filer.sync): scope filesystem key sanitization to the local sink (#9894)
* fix(filer.sync): scope filesystem key sanitization to the local sink destKey ran every sink key through escapeKey, whose Windows build strips colons. Colons are illegal in NTFS filenames so the local sink needs that, but s3/filer/azure/gcs/b2 accept them as ordinary key bytes — stripping them silently diverged the destination key (a source a:b replicated as ab). Move the sanitization into the local sink behind a Windows build tag, applied at every entry point so the previously-unescaped in-place-update paths stay consistent. Non-local sinks now keep the raw key; non-Windows builds are unchanged; a leading drive-letter colon is preserved. * test(filer.sync): cover incremental destKey and localsink update/delete sanitization Lock the colon-preserving behavior for the incremental destKey branch, and extend the Windows local-sink test to assert UpdateEntry and DeleteEntry also sanitize the key, not just CreateEntry. |
||
|
Chris Lu
|
ed470dccb1 |
mini: grow volumes one at a time
Mini auto-sizes a few large volume slots, but the master pre-grows 7 volumes per new collection. Under a filer group each S3 bucket is its own collection, so the first buckets claimed every slot and later writes failed to assign a volume. Cap mini's volume_growth copy counts to 1. |
||
|
|
1b5f1c1f3b |
feat(filer.backup): -initialSnapshot re-seeds a reinitialized destination (#9828)
* feat(filer.backup): add -resetCheckpoint to force a fresh sync filer.backup resumes from a per-sink offset persisted in the source filer's KV. There was no first-class way to discard that checkpoint and re-run from the beginning short of guessing a large -timeAgo, which also skips -initialSnapshot. Add -resetCheckpoint: before reading the offset, write 0 for this sink so getOffset returns 0, isFreshSync stays true, and -initialSnapshot re-runs a full walk. Effective only when -timeAgo is 0. The flag is cleared after the first successful reset: runFilerBackup retries doFilerBackup forever on error, so leaving it set would re-zero the checkpoint on every retry and never make forward progress after a transient failure. Later retries resume from the persisted checkpoint instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(filer.backup): keep fresh-sync intent when offset read fails after reset After -resetCheckpoint writes offset 0, a transient getOffset read-back error flipped isFreshSync to false, which skipped the -initialSnapshot walk the reset explicitly requested. Track that the reset happened this iteration and, on a getOffset error, preserve isFreshSync=true in that case (the non-reset path keeps treating a read error as "not fresh" to avoid re-walking on transients). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(filer.backup): skip offset read-back on reset instead of tracking a flag Replace the didReset bool by branching: on -resetCheckpoint, clear the offset and start fresh without reading it back (we just wrote 0, so the state is known); otherwise read the offset as before. This drops the redundant getOffset RPC after a reset and removes the read-back error case entirely, so no separate flag is needed to preserve isFreshSync. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * filer.backup: -initialSnapshot re-seeds on every start; drop -resetCheckpoint -initialSnapshot now walks the live tree whenever -timeAgo is 0, seeds the destination, and overwrites the saved checkpoint, rather than running only on a fresh sync. That re-seeds a reinitialized destination on its own, so the separate -resetCheckpoint flag is gone. The walk runs once per process: the in-memory flag is cleared after the watermark is persisted, so the retry loop resumes from the persisted checkpoint instead of re-walking on every transient error. A process restart re-walks, so remove the flag once the backup is caught up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
89cbb1c558 |
admin: default -dataDir to "." so maintenance task state persists across restarts (#9856)
admin: default -dataDir to "." so maintenance task state persists Previously -dataDir defaulted to empty, so the admin ran maintenance in memory only: task state was never saved and maintenance tasks (notably EC balance/rebuild) were re-issued every scan cycle without converging, churning EC shards (moves landed shards without their .ecx index, leaving EC volumes unloadable/missing shards). Default -dataDir to "." (the process working directory, which under the standard systemd unit is the admin's data dir) so state persists out of the box. |
||
|
Chris LuandGitHub
|
755af4adf4 |
s3: actually bind outbound connections when -ip.bind is set (#9849)
* s3: set outbound bind IP before the first filer dial Standalone weed s3 dialed the filer for GetFilerConfiguration before SetOutboundLocalIP ran, so that gRPC conn was created with the stock dialer and no source address. gRPC caches conns by address and reuses the original dialer on reconnect, so the s3->filer connection kept leaving from the OS-chosen source for the life of the process even after the bind IP was set a moment later. * grpc: install the outbound-bind dialer unconditionally The dialer was installed only when OutboundLocalAddr was already set at GrpcDial time, baking the source-address decision into the cached conn, so a conn dialed before the bind IP was configured never bound. Install the context dialer always and decide per dial: bind through OutboundDialContext once a source is set, otherwise fall back to the stock net.Dialer so default deployments keep gRPC's dial timeout and keepalive behavior. The bind now applies on the next reconnect regardless of ordering, matching the HTTP transport's unconditional DialContext. |
||
|
Chris LuandGitHub
|
be7f417a03 |
ip.bind: bind outbound connections to the configured address (#9834)
* ip.bind: bind outbound connections to the configured address -ip.bind only governed listeners; outbound gRPC and HTTP connections let the OS pick the source IP, which may not even be able to reach the target. Mirror the bind address into a process-global source address and apply it to outbound TCP dials: the gRPC context dialer, the per-client HTTP transports, and the default transport. Loopback targets and unix sockets keep the OS-chosen source so same-host traffic still works. * ip.bind: first-write-wins source IP, skip on address-family mismatch Make SetOutboundLocalIP first-write-wins so a `weed server` component's own bind setting (run in its goroutine) can't clobber the process-wide source address the top-level -ip.bind already established for the other components. Skip source binding when the target is a literal IP of a different family than the bind address, since forcing a mismatched source fails the dial. |
||
|
Chris LuandGitHub
|
ab7be7867d |
security: hot-reload JWT signing keys on SIGHUP (#9826)
* security: reload JWT signing keys on SIGHUP Signing keys were read once in the server constructors and never refreshed. After a key rotation (Secret update, divergent reads) the in-memory key stayed stale and every request kept failing "wrong jwt" until the affected process was restarted. Add Guard.UpdateSigningKeys and call it from the master, volume and filer reload paths and the s3 reload hook, next to the existing whitelist refresh. Make the global chunk-read JWT cache reloadable via an atomic swap, and register the master's Reload with grace.OnReload -- it was never wired, so the master ignored SIGHUP entirely. Mirror the same refresh in the Rust volume server's SIGHUP handler. * security: swap signing keys behind an atomic pointer Addresses review feedback on the in-place key swap: SigningKey is a []byte, so reassigning the Guard fields while a request handler reads them is a data race that can tear the multi-word slice header and read out of bounds. Hold the four signing-key fields in an immutable signingConfig snapshot behind atomic.Pointer; UpdateSigningKeys swaps the whole pointer, so a reader sees either the old keys or the new ones. Reads go through new SigningKey/ExpiresAfterSec/ReadSigningKey/ReadExpiresAfterSec accessors. The Rust guard is already safe: every read and the SIGHUP write go through the shared RwLock<Guard>. * security: fold whitelist + auth state into the atomic snapshot Review follow-up. UpdateSigningKeys still wrote isWriteActive while the request path read it (and the whitelist maps) unsynchronized, so a SIGHUP under load could expose an inconsistent mix of activation bits and whitelist contents. Move all hot-reloadable Guard state -- keys, expirations, whitelist, and the activation flags -- into a single immutable guardState swapped behind one atomic.Pointer. The Update* methods take a small mutex to serialize the read-modify-write; readers stay lock-free. The concurrency test now also rotates the whitelist and probes IsWhiteListed under -race. Also read each signing key once per branch in the volume/filer JWT auth checks, so a reload landing mid-check can't take the allow-fast-path after auth was enabled or verify against a different key than the branch saw. |
||
|
|
6e8002f065 |
fix: handle meta backup offset errors safely (#9818)
* fix: log meta backup offset errors * fix: log meta backup offset errors * fix: exit on meta backup offset errors Exit with a non-zero status when the initial metadata backup offset cannot be persisted. Classify offset-read failures during streaming so the backup process exits instead of retrying forever, allowing supervisors to restart and bootstrap from a missing checkpoint. * meta backup: read offset in the loop, drop offset error type Reading the saved offset inside the retry loop makes an offset read failure a clean exit and a stream error a retry, without a typed error to tell them apart. streamMetadataBackup now takes the start time. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
 Fabian HardtGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Chris Lu |
ce6a51468a |
sftpd: support SSH user certificates signed by a trusted CA (#9815)
* sftpd: support SSH user certificates signed by a trusted CA Adds a new "certificate" auth method to weed sftp. When enabled, the server loads trusted CA public keys from -trustedUserCAKeysFile (OpenSSH authorized_keys format, one or more keys) and accepts only ssh.Certificate blobs of type UserCert on the public-key channel. Validation uses ssh.CertChecker: CA signature, ValidAfter/ValidBefore, non-empty ValidPrincipals and SSH login user must appear in ValidPrincipals. The authenticated user must exist in the user store; home dir and permissions resolve as before. Behaviour mirrors MinIO's --sftp=trusted-user-ca-key and OpenSSH's TrustedUserCAKeys: when certificate auth is active, plain (non-cert) public keys are rejected even if "publickey" is also listed. Default authMethods remain "password,publickey", so existing deployments are unaffected. * Update weed/sftpd/auth/certificate.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * sftpd: address review feedback on certificate auth - Pre-marshal trusted CA public keys in IsUserAuthority instead of re-marshaling on every authentication attempt (gemini-code-assist). - Differentiate user-not-found from underlying store errors via errors.As(*user.UserNotFoundError) so backend/read failures are no longer reported as bad credentials (coderabbitai). - Fix the corresponding sanity check in the missing-file test to use errors.As instead of errors.Is (UserNotFoundError has no Is method, so the previous check never matched) (coderabbitai). * sftpd: register trustedUserCAKeysFile flag in filer and server commands The new field on SftpOptions is dereferenced unconditionally in resolvePaths(), but only the standalone `weed sftp` command was wiring its flag. `weed filer` and `weed server` both embed an SftpOptions value and call resolvePaths() on it, so they hit a nil pointer dereference at startup. Register `-sftp.trustedUserCAKeysFile` in both commands and update the -sftp.authMethods help text to mention the new "certificate" method. Fixes the SFTP Integration Tests CI failure on this PR. * helm: expose SFTP certificate auth in the SeaweedFS chart Adds Helm-chart support for the new SSH user-certificate auth method: - values.yaml (sftp:) gains `trustedUserCAKeys` (inline OpenSSH authorized_keys-format CA public keys) and `existingCAKeysSecret` (reference an externally managed Secret). Same pair added under allInOne.sftp with a null default that falls back to the top-level sftp.* setting. - New template templates/sftp/sftp-ca-secret.yaml renders a chart-managed Secret <release>-sftp-ca-secret with `ca_user.pub`, but only when SFTP is enabled, "certificate" is in authMethods, inline keys are provided, and no existingCAKeysSecret is set. - templates/sftp/sftp-deployment.yaml and the all-in-one deployment template add `-trustedUserCAKeysFile=/etc/sw/sftp_ca/ca_user.pub` to the weed sftp command, mount the CA secret at /etc/sw/sftp_ca and add the corresponding volume. All cert-auth bits are guarded by `contains "certificate" authMethods` so existing users see no change. - authMethods help text updated to mention "certificate". Verified end-to-end on a local k3d cluster: cert login succeeds, plain-pubkey login is rejected with "public key without certificate not allowed". * helm: fail render when SFTP certificate auth lacks CA keys When certificate is in authMethods but neither trustedUserCAKeys nor existingCAKeysSecret is set, the deployment mounted a secret that the chart never renders, leaving the pod stuck on a missing volume. Fail at template time with a clear message instead. * sftpd: fix stale auth-method list in SFTPServiceOptions comment keyboard-interactive was never implemented; certificate is the new supported method. Match the CLI help text. * sftpd: test Manager wiring of certificate vs public-key channel Cover the channel takeover at the Manager level: certificate auth displaces plain public-key auth when both are enabled, public-key auth stays put otherwise, and enabling certificate without a CA file errors. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
 
|
e3e02d3364 |
[CheckDisk]: implement disk health detection (#9560)
* [CheckDisk][GRPC]: implement MVP for disk health detection, added timeout for new grpc connections * fix(volume): build disk health check on every platform setDiskStatus only existed behind the statfs build tag, so disk.go failed to compile on windows, openbsd, solaris, netbsd and plan9. Move the timeout wrapper and failure tracking into the shared disk.go and have each platform's fillInDiskStatus return an error, so every platform gets the same protection from a stuck filesystem. Also restore the uint64(fs.Bavail) cast: Bavail is int64 on freebsd, so the unguarded multiply broke the freebsd build. * fix(volume): keep one outstanding statfs probe per disk A stuck statfs used to leave isChecking cleared by the timeout path, so the next check spawned another goroutine while the previous one was still blocked in the syscall, leaking one goroutine per minute on a hung disk. Clear the flag only when statfs returns and treat an overlapping check as a failure, so a hung filesystem keeps a single outstanding probe and still gets reported. * fix(volume): assume disk available until the first health check isDiskAvailable defaulted to false, and CollectHeartbeat skips locations that are not available. A freshly started volume server would therefore omit every volume from its first heartbeats until the async CheckDiskSpace ran, so the master could briefly treat all of them as missing. * fix(volume): label the disk error metric by data directory The new gauge tagged the series with IdxDirectory while every neighbouring resource gauge uses Directory, so the error series would not line up with them in dashboards. Also log the underlying error instead of a generic message. * test(volume): cover disk health success and repeated-failure paths * fix(volume): make a healthy disk the zero-value default Track the disk as isDiskUnavailable instead of isDiskAvailable so the safe state is the zero value, matching isDiskSpaceLow. CollectHeartbeat only skips a location once a check has actively marked it unavailable, so any DiskLocation built without running CheckDiskSpace (tests, future call sites) still reports its volumes instead of silently dropping them. * feat(disk): detect degraded disks using IO latency probes * feat(stats): introduce configurable disk I/O health probe with EWMA-based latency detection * feat(disk): replace EWMA with sliding window algorithm for disk health detection and added user-friendly options * feat(disk): improve disk health probing and recovery * feat(volume): configure disk health checks via volume.toml * fix(volume): Remove disk IO probe CLI options --------- Co-authored-by: ptukha <ptukha@tochka.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |