9527 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandGitHub	a9de90ae29	test: wait for every queued flush before deleting the log files (#10546) ... TestSubscribeLoop_FlushProvenGapSkipsToRetained deleted the log files once the eviction watermark's own flush had landed, while the windows sealed after it were still queued. Those flushes then wrote their files back, and the subscriber served them from disk instead of taking the gap-skip path, so the windows whose files really were gone came out missing. Wait through the last sealed window instead.	2026-08-03 09:26:42 -07:00
Chris LuandGitHub	4f692bf9c3	mount: build the package on windows (#10535) ... * mount: drop the unused go-fuse fs package dependency WFS embedded fs.Inode but never used any of its methods, and the only other reference was RENAME_EXCHANGE, a constant sitting next to three literals. Removing both drops fs and five internal packages from the mount build graph. * mount: build the package on windows Windows has no fcntl lock types, no O_ACCMODE and no x/sys/unix, so a handful of constants kept weed/mount pinned to unix even though the code using them is portable in-memory logic. Route them through per-OS shims and give setBlksize a windows no-op. The POSIX lock table now compiles on windows but stays unreachable: WinFsp resolves byte-range locks in its own kernel driver, so nothing will feed it there. go.mod points at a go-fuse branch commit and needs repinning to a release tag once that lands. * ci: cross-compile for windows Nothing caught the unix-only constants creeping into weed/mount until a release build failed. * mount: let readdir feed a sink instead of the kernel buffer doReadDirectory wrote directly into fuse.DirEntryList, which is the kernel's wire format. A front end that is not the kernel would have to pack entries only to parse them straight back out. Route it through DirEntrySink instead. ReadDir and ReadDirPlus pass the reply buffer, so nothing changes for the FUSE server. * mount: pin go-fuse v2.9.4 for the windows build	2026-08-03 01:07:49 -07:00
Chris LuandGitHub	529ffa5c86	mount: drop the unused go-fuse fs package dependency (#10534) ... WFS embedded fs.Inode but never used any of its methods, and the only other reference was RENAME_EXCHANGE, a constant sitting next to three literals. Removing both drops fs and five internal packages from the mount build graph.	2026-08-03 00:53:31 -07:00
4 people	813a4b0711	fix(filer): stop skipping recent unflushed events on metadata subscription gaps (#10501) ... * fix(filer): don't skip unflushed events on metadata subscription gaps A subscriber that falls behind the in-memory log ring during a write burst could have its read position jumped past events that were evicted but not yet flushed to disk — silently lost for filer.backup/filer.sync/ mount subscribers. Route all three gap-skip sites through resolveDiskGapResume: only skip past windows older than a settled horizon (2*LogFlushInterval); recent gaps wait for the flush and re-read disk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filer): harden metadata gap-skip guard Address review findings on the settled-horizon guard: - local subscriptions: gate the skip on the buffer's flush watermark observed before the disk read (resolveLocalGapResume) — a disk miss is then proof the gap is empty, with no wall-clock assumptions - aggregated subscriptions: cap skips strictly below the horizon boundary (persisted reads exclude ts <= cursor) and pace capped advances, so the sliding horizon cannot cause disk-probe spinning - replace unbounded sync.Cond waits with a bounded select on the buffer's subscriber channel + retry timer + ctx cancellation, eliminating the lost-wakeup stall Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filer): close remaining gap-skip loss paths from re-review - log_buffer: a sentinel-offset (time-based) read below the earliest in-memory entry silently started at earliest, skipping a window that may hold evicted-but-unflushed events. Track the ring's eviction watermark (lastEvictedTsNs) and keep the inclusive fast path only when nothing at/after the position was ever evicted; otherwise return ResumeFromDiskError so the subscription gap guard decides. - capped horizon advances stay on the disk-probe path (never expose a mid-gap position to the memory read) and keep pacing - gap jumps land just below earliest: positions are exclusive, so the earliest entry itself is still delivered - subscriber notification keys include clientId/epoch so a replacement stream never inherits a channel the old stream's cleanup closes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(log_buffer): generalize the eviction-watermark read gate Third-review findings: - epoch/zero-time reads bypassed the eviction watermark: gate them the same way, so a SinceNs=0 subscriber cannot silently start at the earliest retained entry after an unflushed window was evicted - apply the watermark gate regardless of the cursor's batch offset (batch offsets carry no meaning for time-based reads); this also serves adjacent cursors (earliest == current+1) from memory instead of stalling them in the gap loop - LoopProcessLogData reader names include clientId/epoch, since they are registered as subscriber keys internally (same collision as the outer notification keys) - test: pin the gate (below/at watermark, epoch-after-eviction) and update the slow-consumer test to the sharper contract — complete in-memory history is served from memory; disk only once evicted Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(log_buffer): watermark equality is unsafe for inclusive sentinel cursors Sentinel (Offset <= 0) time-based cursors search from ts-1ns, i.e. they read inclusively of their own timestamp — and the evicted window may end exactly at that timestamp. Allow watermark equality only for exclusive (positive-offset) cursors; sentinel cursors must be strictly above it. Also shut down the test buffer and pin the equality cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filer): wait on an empty aggregated buffer instead of falling through When a disk read finds nothing for a ResumeFromDiskError gap and the aggregated buffer has no readable entries (zero earliest time), resolveDiskGapResume declines to advance and control fell through to the in-memory read — which returns ResumeFromDiskError again immediately, spinning through the probe cycle without any wait. Fold the case into the existing recent-gap branch so it waits (notification, cancellation, or the retry interval) before re-probing, matching the local subscription path, which already waits unconditionally. * fix(filer): serve the exact eviction-boundary entry without another flush wait When the flush watermark has passed the earliest in-memory entry but that entry sits exactly one nanosecond above the cursor, the exclusive resume target collapses onto the cursor and resolveLocalGapResume declines to advance — while a sentinel cursor at the eviction watermark keeps deferring the in-memory read to disk. Progress then depends on the next flush cycle. Re-arm the cursor with a positive (exclusive) offset instead: ReadFromBuffer explicitly allows positive-offset cursors at the watermark, so the boundary entry is served from memory immediately. Also promote the aggregated path's horizon-capped gap skip to a warning: that skip may pass events a stalled flush lands later (the aggregated ring has no flush watermark to gate on), so operators should see when a flush stall outlasts the settled horizon. * fix(filer): resume a gap skip with an exclusive cursor The resume position landed one nanosecond below the earliest in-memory entry but kept the inclusive sentinel offset. When that timestamp is also the eviction watermark, the read gate answers ResumeFromDiskError for an inclusive cursor, the disk read finds nothing, and the resume target collapses onto the cursor, so neither helper can advance: the subscriber parks on timed waits forever. A skip is only taken once the gap is proven empty, so nothing remains to deliver at the resume timestamp. Resume exclusively instead, and let an inclusive cursor already sitting on the target count as progress. * fix(filer): gate aggregated gap skips on eviction, not wall clock Wall-clock age never proved persistence. The aggregated ring has no flush watermark because peers persist their own local logs, so a horizon of 2 * LogFlushInterval was standing in for one. While the disk stayed empty the horizon kept sliding forward and walked the cursor past evicted-but-unflushed events one window at a time - the volume-outage stall this change set exists to survive. The ring does carry a real proof: its eviction watermark. If nothing at or after the cursor was ever dropped, memory still holds every entry after it and the gap is provably empty. Below the watermark entries were dropped and only the producing peer's flush can supply them, so wait for that flush instead of advancing. The skip that remains is the one the ring can prove, which keeps the infinite-loop guard for a genuinely empty gap. * perf(filer): stop re-probing disk on every metadata append A subscriber parked on a gap woke on the log buffer's subscriber channel, which fires on every append. On the aggregated path each wake re-ran a full ReadPersistedLogBuffer - a ListDirectoryEntries plus a readahead goroutine - so one parked subscriber turned every cluster-wide metadata event into a store query, exactly while the cluster is already struggling with the flush stall that parked it. An append also cannot settle the gap: what this waits on is a peer persisting its own log, which nothing local signals. Wait on the timer alone there. The local buffer does signal its flush on that channel, so keep it, but drain the stale token first: otherwise the appends riding the same channel spin the wait at write rate. The retry interval covers the notification the drain discards. * fix(filer): surface a metadata subscriber parked on a gap Refusing to skip an unresolved gap trades silent loss for a silent stall, and a stall is no easier to diagnose: filer.sync and mount followers just stop advancing, with no error on either end. The only trace was a V(3) line nobody runs with. Warn on entry and once a minute after, and carry a subscribe_gap_stalled gauge for the duration, so a flush that never lands shows up as a stalled subscriber rather than a consumer that mysteriously went quiet. * refactor(filer): drop the metadata listener cond with no waiters left Both listenersCond.Wait() sites are gone, so listenersWaits never leaves zero, the guard in the filer's notify callback never fires, and the two Broadcast calls around client registration wake nobody. Remove the cond, its counter and its lock, and pass a nil notify func: the log buffer already skips a nil one, and the subscriber channels carry these wakeups now. * test(log_buffer): drive the eviction watermark through a real seal Both eviction tests wrote lastEvictedTsNs directly, leaving the one line that sets it uncovered: copyToFlushInternal has to read slot 0 before SealBuffer shifts it out, and moving that read one statement later still passed every test. Append past the ring instead, and check the watermark appears only on the seal that drops a window, matches that window's stop, and then advances. The buffer under test never flushes, matching the aggregated meta ring where the watermark is the only emptiness proof. * refactor(filer): build the subscriber reader name once per stream It was rebuilt on every loop iteration from values that cannot change for the life of the stream. Hoist it next to the notification key it mirrors. * docs(filer): tighten the gap-handling comments Several ran to eight lines restating the same reasoning at each site. Keep the non-obvious why, drop the retelling. * fix(filer): mark a confirmed disk position as an exclusive cursor A disk read returns the timestamp of the last entry it handed to the subscriber, but the cursor built from it stayed inclusive. Land that cursor exactly on the eviction watermark and the read gate sends it back to disk for an entry the disk just delivered; the re-read finds nothing, neither emptiness proof holds for an inclusive cursor there, and the subscriber parks - until some later flush, or forever while one stays stalled. Everything after the watermark was sitting in memory the whole time. Carry the offset instead, so it also covers the ring evicting onto the cursor after the read rather than before it. The constant is no longer gap-specific, so it is now named for what it asserts. * perf(filer): re-park a gap wait woken by an ordinary append Draining one stale token did not bound anything: the subscriber channel carries an append per metadata write, so under continuous writes the next one satisfies the wait immediately. During the write burst these stalls come from, each parked subscriber ran the recovery loop at write rate rather than the intended two-second cadence. Only a flush can settle a gap, so check the flush watermark on wake-up and go back to waiting if an append is all that arrived. The retry timer is created once, so re-parking does not extend the interval. * fix(filer): keep a disk-derived cursor inclusive Marking every disk position exclusive assumed the entry at that timestamp was the only one there. On the aggregated stream it is not: disk can deliver one filer's persisted event at T while another filer's event at the same T is still unflushed in the aggregate ring. The exclusive cursor skipped it, and since the persisted reader also excludes ts <= its start, nothing would ever bring it back. Take the weaker guarantee instead. The reason the exclusive cursor was introduced - an inclusive one landing on the eviction watermark parks forever - is better answered in the resolvers: at the watermark memory holds nothing (retained windows start strictly after it) and the persisted reader cannot return that entry at any later time either, so refusing the gap buys nothing and never ends. Skip it whatever the cursor's inclusivity. That proof does not depend on a flush, so the local resolver takes it as a second, independent disjunct alongside its flush watermark. * perf(filer): wake a parked gap wait on flushes only Re-parking on an append bounded the work but not the wake-ups: the subscriber channel carries one per metadata write, so a parked subscriber still took a scheduling round-trip per write. Worse, draining it kept the channel empty, so every writer's non-blocking notify succeeded instead of falling through - the burst paid for the wake-ups too. Give the log buffer a flush-only subscriber list, notified from loopFlush, and park on that. The append channel now fills once and stays full, which is exactly the state the non-blocking send is designed for. * fix(filer): stop the gap-stall gauge from leaking a series per connection clientName is req.ClientName + "@" + peer address, so it carries the client's ephemeral source port and changes on every reconnect. Labelling the gauge with it minted a new series per connection, and clearing it only ever Set(0), so nothing was ever released - a client in a reconnect loop grows the filer's metric map and /metrics payload without bound. Key it on the stable client-supplied name, as the neighbouring subscribe gauge already does, and delete the series on teardown. Two logging fixes ride along, both in the same reporter: the resume warning was unpaced while the park warning throttles to one a minute, so a burst that parks and resumes every couple of seconds warned on every cycle; and clear() doubled as the teardown path, announcing "resumed after 14m0s parked" for a client that actually gave up and disconnected still behind. * fix(filer): give a parked gap wait the exits the read loop has A park never re-enters the read loop, so every exit that loop relies on stopped working while a subscriber was parked. It kept scanning the filer store every 2s for a client that a higher-epoch reconnect had already superseded, until the TCP connection finally died - hours, on a half-open one. It never reached the only code that honors UntilNs, so bounded callers like `weed shell fs.verify` and `filer.meta.tail -until=` hung instead of exiting. And a notification channel closed out from under it turned the bounded wait into a spin, since a receive on a closed channel returns instantly. Check all three where the subscriber actually waits. Bound the wait too: waiting is productive while the window is still queued for flush, but a peer that never returns, or whose filer store this filer cannot read at all, makes it permanent - and a subscriber that silently stops delivering is no better than one that silently skips. Fail the stream after that instead of hanging, loudly enough to say which gap and for how long. * fix(log_buffer): keep the eviction gate out of the shared read path The gate belonged to the filer's subscribe loops but was installed in ReadFromBuffer, which the message queue shares and which has no gap handling of its own. Three MQ paths broke on it: GetUnflushedMessages asks for everything in memory past the flush watermark and got ResumeFromDiskError instead, so SQL results silently dropped unflushed rows; a RESET_TO_EARLIEST consumer's epoch cursor was sent to disk, and the MQ disk reader resets an empty read back to epoch, so the whole partition replayed on every pass; and a disk cursor landing on the watermark carried offset -2, which the gate refused and the disk reader's ts <= start filter also skips, so neither side could ever serve it. The rewrite also dropped the old Offset <= 0 requirement, letting a stale positive-offset cursor jump to memory over on-disk history, and refused any negative-timestamp cursor even with nothing evicted. Restore the read path exactly as it was and keep only lastEvictedTsNs, which is the useful primitive. The filer loops now consult the watermark themselves before reading memory, which is where the gap handling that makes the refusal actionable already lives - and they check it every pass, not just when the disk came up empty, since a disk read can leave the cursor short of the watermark too. * fix(filer): read the log file whose window spans past its own name A log file is named for the start of the window it holds, but a window runs up to a flush interval longer, so "12-30" can hold entries through 12:31:58. File selection compared the cursor's minute against that name and skipped anything sorting earlier, so a subscriber resuming at 12:31:10 never saw the rest of that file: the read reported nothing on disk while the entries sat in it. That was survivable when a miss only meant "wait and retry", but the gap resolvers now read a miss as proof the range is empty and move the cursor past it, which turns those entries into silent loss. Start the file scan a flush interval early; entries are still filtered against the exact cursor, so this only opens one more file and never re-delivers. * fix(filer): resume a gap with a cursor the memory read will serve Reverting the eviction gate put ReadFromBuffer back to refusing every positive-offset cursor below the in-memory window, but the resolvers still handed back one - earliest-1 marked exclusive. So the resume bounced straight to ResumeFromDiskError, the disk had nothing, and the resolver saw its own cursor as no progress and parked: a subscriber stalled, and after the new bound failed outright, with the whole gap sitting in the ring the entire time. The exclusive marking only existed to dodge a park the gate itself caused, and the gate is gone. Resume at earliest with the sentinel offset, which is the position master used and which case 2.1 reads inclusively. That makes the resume unconditionally ahead of the cursor, so the progress check it needed goes away with it. * fix(filer): stop dropping a log file whose window outruns its name Listing the earlier file was not enough: the iterator then decided whether to read it by comparing the *following* file's name against the cursor, which treats that name as an upper bound on this file's contents. It is not one. A file is named for the start of the window it holds, minute-truncated, and the window runs up to a flush interval longer, so "12-30" can hold an event at 12:31:20 while "12-31" sits right after it - and a cursor at 12:31:10 skipped straight past the event. Bound the decision on the file itself: skip it only when its name plus the minute truncation plus a flush interval still lands at or before the cursor. That also covers the last file in the queue, which the old check never skipped because it had no successor to compare against. * fix(filer): stop a chunk-ref read from rewinding the subscriber CollectLogFileRefs reports the minute-level name of the last file it shipped, and the caller assigns that straight to the read position. Since the scan now reaches back a flush interval to catch a spanning file, a request at 12:31:10 that picks up the 12-30 ref moved the cursor to 12:30:00 - so the memory read that followed replayed events older than the client's own SinceNs and re-sent what the chunk reader had already been handed. The same rewind was reachable before, within a minute, whenever the last file's name sorted behind the request. Clamp the reported position to the one that was asked for. It still under-advances by design, since the server never reads the entries it ships refs for and cannot know where they end. * fix(filer): resume below earliest so a one-entry window is not skipped Moving the resume onto earliest itself was wrong for the smallest window there is. A sealed window holding a single entry has startTime == stopTime, and the sealed-buffer lookup only enters a window whose stopTime is strictly after the cursor, so a cursor sitting exactly on earliest walks past it and its sole event is never delivered. Low-volume metadata windows are routinely one entry, which is precisely when losing it is hardest to notice. Go back to one nanosecond below, which takes the startTime.After branch and returns the whole window, and keep the sentinel offset the memory read requires. The earlier test used an active multi-entry buffer, where both cursors happen to work; the new one seals single-entry windows and compares which entry comes back. * fix(filer): count a persisted read as progress only when it moves A chunk-ref read reports the minute-level name of the last file it shipped, now clamped so it never rewinds, so it comes back non-zero even when it names the position the subscriber already held. The loops read non-zero as progress: they cleared the stall timer, then found the cursor still short of the eviction watermark and parked again. Every retry re-shipped the same refs and reset the timer, so the bound that is supposed to end an unrecoverable stall was never reached - and a chunk-capable client buffers those refs waiting for an event that never comes, so it just accumulates duplicates. Require the reported position to be strictly ahead of the cursor. * fix(filer): count the evicted ranges the aggregated stream cannot prove The eviction watermark belongs to the merged ring, but the disk it gets checked against is the union of every peer's own log and each peer flushes on its own schedule. A read that lifts the cursor from below the watermark to above it may have done so entirely on a peer that is already ahead, while a lagging peer still holds unflushed events inside the range just crossed; when it flushes them they sit behind the cursor and are never delivered. One aggregate maximum is not proof that every peer persisted the range. Nothing available locally separates that from the ordinary case where every peer had in fact persisted it: the aggregator tracks peers by address while log files carry a random per-filer id, so "has this peer flushed through T" cannot be answered here at all. Deciding it needs the source filer's own flush watermark carried on the subscribe stream, which is a wire change this does not make. Count and log the crossing so the window is at least measurable instead of invisible. * fix(filer): send log file refs through the pipelined sender sendLogFileRefs wrote on the raw gRPC stream while pipelinedSender's goroutine concurrently calls Send for queued memory events - two senders on one stream, which gRPC forbids. The window used to open once per fall-behind; the gap retry loop now reopens it every pass. Routing refs through the sender also restores ordering: refs used to overtake up to 1024 queued events, and the client treats any non-ref message as the signal to process buffered refs, so overtaken refs were applied against the wrong position. The reason refs bypassed the sender was the batcher: their TsNs of 0 reads as far behind, and the client recognizes refs by the top-level field alone - a refs envelope would drop its Events tail, and refs inside Events would be applied as an empty event. Teach the batcher instead: refs messages always go solo, and one drained mid-batch is sent solo right after that batch. * fix(filer): count parked subscribers instead of flagging them by name The per-client gauge series could not work. Its label was rebuilt from the peer address at first, which leaks a series per reconnect; keyed on the client-supplied name instead, it collides - every mount registers as "mount" - so one stream's teardown deleted a parked sibling's live series, and the sibling never re-created it because its own park state said the gauge was already set. Either way the alert this gauge exists to drive goes dark. A count needs no identity: Inc on park, Dec on resume or teardown, scope as the only label. Client details stay in the logs. Also start warning only once a stall has outlived the warn interval. park() warned immediately on every first park, so catch-up churn that parks and resumes every couple of seconds logged a warning pair per cycle - exactly the flood the pacing was supposed to prevent, burying the long-stall warnings that matter. * fix(filer): one gap resolver, and re-arm an unservable adjacent cursor The two resolvers were the same function - the aggregated one is the local one with a flush watermark of zero, since its ring never flushes - duplicated down to the comment justifying the resume target. A fix applied to one and not the other is how the two streams drift; merge them. The merge carries the one behavioral fix both copies needed. Timestamp collision bumps make adjacent entries exactly 1ns apart, so an entry ending an evicted window leaves the cursor exactly one below earliest with a positive batch offset. The resume target then equals the cursor and both copies refused it as no progress - but that cursor cannot be served (ReadFromBuffer refuses positive offsets below the window) while the sentinel resume at the same timestamp is, and both deliver exactly the entries after it. Refusing parked a subscriber whose data was entirely in memory: until the next flush locally, and through a 15-minute stall failure on the aggregated path. Advance on equal target when the held cursor is exclusive; a sentinel there is already served, so it still refuses. * fix(filer): a bounded subscription parked exactly on UntilNs is finished The park's UntilNs check was strict while the bound is inclusive and cursors are exclusive: a disk read whose last entry sits exactly on UntilNs leaves the cursor there with everything up to the bound already delivered. If the next range was an unprovable gap, the completed subscription parked anyway and eventually failed - fs.verify hanging and then erroring on a healthy cluster. * fix(filer): re-derive the subscribe loops as one state machine The loops had grown three generations of gap handling - a post-disk-read branch, a post-memory-read branch, and an every-pass guard bolted on in front of the memory read - each consulting state the others mutated. The worst interaction wedged the aggregated stream permanently: diskExhausted compared the disk result against the cursor that result had just updated, and paired it with a ResumeFromDiskError latch that only a resolver advance cleared, so one fall-behind sent every later pass into the gap block and LoopProcessLogData never ran again. Mounts kept a healthy-looking stream and applied nothing. Both loops now run the same derived sequence. One disk pass; progress is the pre-update cursor against the result, freshly each pass. One gap decision before the memory read: a cursor the ring evicted past either keeps draining the disk (it just advanced), resolves forward (the gap is proven empty), or parks - and a cursor memory refused with nothing evicted after it re-arms onto the retained window. The stale-latch branch is gone: the error is only consulted on a pass whose own disk read came up empty. All four parks go through one parkOnGap helper, so the exits live in one place. The eviction watermark is read after the disk read and the same value feeds both the guard and the unproven-crossing report, which previously compared against a snapshot taken before a potentially minutes-long backlog read and missed evictions landing during it. The next-day jump now clears the stall reporter - it used to leave a stale park epoch that could kill the next brief park instantly at the 15-minute bound - and counts its own watermark crossing. Chunk-ref reads get two rules the old loops lacked. Refs are sent once per position: retries re-sent identical batches every two seconds into a client that only drains them on a non-ref message, growing an unbounded pending list. And when refs cannot advance the cursor - they report file start minutes, which sit below the content the client was actually given, pinning the cursor under the watermark forever during bursts - the pass falls through to entry reads, which move the cursor by real timestamps and double as the client's drain signal. * fix(filer): give up on an unprovable gap instead of failing the stream Failing after maxGapStall assumed the client could do something better, but every consumer just reconnects at the same SinceNs and hits the same wall, so an unprovable gap - a dead peer, or a peer whose filer store this filer cannot read at all - turned into a permanent 15-minute fail/reconnect loop delivering nothing. Master handled the same state by skipping instantly and silently. Take the middle: wait the full bound, then abandon the gap and resume at the eviction watermark, where everything retained starts strictly after, so the loss is exactly the range that could not be proven. The skip shares the unproven-crossing counter and logs at error level - loss is bounded, recorded, and the stream keeps working. A stall with nothing evicted past the cursor loses nothing by waiting, so it restarts the clock and keeps parking rather than skipping. * test(filer): pin the file-skip bound through the production predicate The spanning-file test asserted against its own copy of the arithmetic, so a regression in the iterator - restoring the next-file-name comparison or dropping the flush-interval term - would keep CI green while re-introducing the silent loss the fix closed. Extract the bound into logFileMayContainAfter, call it from the iterator, and point the test at it; breaking the production expression now fails the test. * test(log_buffer): pin the flush-subscriber contract The registry the filer's gap parks wait on had no test at all. Cover the observable contract: an append never wakes a flush subscriber, a flush does with the watermark already stored, unregistering closes the channel so an abandoned waiter unblocks, and double or unknown unregisters are harmless. The store-before-notify ordering in loopFlush is what makes the parks' wake-up re-check sound, and it is not black-box testable - reordering leaves a same-goroutine window of nanoseconds that hundreds of tight round-trips never catch. Mark it load-bearing at the site instead; a reorder now at least has to argue with the comment it deletes. * fix(filer): a -1 SinceNs is a position, not the refs-gate sentinel The once-per-position refs gate used -1 as "never sent", but a client may legally subscribe with SinceNs=-1, whose cursor timestamp is exactly -1: the very first pass then believed refs were already sent there and fell back to streaming the whole persisted history entry by entry - the bootstrap load chunks mode exists to avoid. Use MinInt64, which no cursor can carry. * refactor(filer): drop the aggregator's listener cond with no waiters left Same shape as the FilerServer cond already removed: nothing increments ListenersWaits and nothing ever calls Wait, so the three Broadcasts wake nobody and the notify callback's guard is always false. Aggregated subscribers wake through the buffer's subscriber channels now. * fix(filer): make the gap metrics say what they count The crossing counter's help text described only the aggregated peer case, but give-ups increment it for local stalls too - a wedged local flush - which sends an operator chasing peer replication when the problem is the local store. Label it by scope and say both. The stalled gauge counted every park, including waits with nothing evicted and nothing at risk, while its help text promised evicted-but-unpersisted events; describe it as what it is, a count of subscribers parked on a gap. * test(filer): make the flush and stall tests assert what they claim The flush-subscriber rounds were vacuous: the probe entry sat above the round timestamps, so every round entry was collision-bumped and the stored watermark exceeded the local value each assertion compared against - the same bump mistake this test suite already made once. Put the probe below the rounds and guard each round against bumping, so a vacuous setup fails instead of passing. The stall-outcome test wrote the reporter's park epoch directly, bypassing the gauge Inc that gaveUp() later Decs - leaving the shared process gauge at -1 for every test that runs after it. Park through the real path, age the park by hand, release what the test holds, and assert the gauge lands back where it started. * fix(filer): finish the stream checks before marking it parked parkOnGap stamped the reporter before waitOnGap ran its instant done exits, so a bounded subscription completing inside a gap state was marked parked for the microsecond before done fired - a phantom gauge blip and a false "disconnected still behind" warning on every healthy completion. Fold waitOnGap into parkOnGap so the done exits run first and the park mark only ever covers a stream that actually waits. While the park owns its timer, back the retry off as the stall ages - 2s probes growing toward one a minute - since every retry re-reads the persisted log, and probing the store each 2s for 15 minutes per parked subscriber during the very outage that parked them makes the bad time worse. The subtest still named for the old fail-the-stream stall behavior goes with the merge. * fix(log_buffer): gate filer cursors against eviction under the read lock The subscribe loops checked the eviction watermark and then read memory, but a seal can land between the two: the read then served a sentinel cursor from the earliest retained window, silently skipping the window just evicted - the loss class this PR exists to make loud, surviving as a race. The only place the check is atomic with the serve decision is inside ReadFromBuffer, under the lock seals take to evict. Rather than put the policy back into the shared read path - which broke four message-queue readers last time - add a new sentinel offset that opts into it: EvictionGatedOffset reads inclusively exactly like -2, except below the watermark it is refused to disk. The filer loops stamp it on every cursor they hand the memory read; a refusal lands in the same gap machinery the loop-side check feeds, so the race collapses into the handled path. MQ cursors never carry it and keep master behavior byte-for-byte. * fix(filer): gate the aggregated gap on received, not bumped, timestamps The aggregated ring rewrites an out-of-order arrival to its head plus a nanosecond, so after any bump-heavy interval - a peer history replay following a restart is enough - its eviction watermark lives above every timestamp that exists on any peer's disk. Comparing a disk cursor against it parked subscribers that had in fact drained every peer's log: a 15-minute delivery freeze ending in a give-up skip and a false loss alarm, on a healthy cluster where master resumed instantly. Track a second watermark in the received timestamp space - the highest pre-bump timestamp among evicted entries - and gate the aggregated loop on that. Disk cursors and received timestamps are the same space, so the comparison means what it says: at or past it, every evicted entry's original was at or below the cursor, and everything flushed of them was already delivered. The bumped watermark keeps guarding the in-ring read gate, whose cursors live in ring space. The local buffer is untouched: it flushes its own bumped timestamps, so there the two spaces are one. * fix(filer): ship each log chunk once and stop echoing ref'd files inline Chunk mode duplicated data through two doors. Consecutive ref collections overlap by design - the scan backs off a flush interval to catch a spanning file, and a filer appends chunks to its newest file - so the same file was shipped again on every pass that re-listed it: the client re-downloaded its chunks, and a duplicated file mid-batch rewinds timestamps inside the client's per-filer merge, which reads each stream as sorted - transiently resurrecting deleted entries during catch-up. Track per subscription how many chunks of each file were shipped and send only the unsent suffix; state prunes with the scan window, so it holds a few files per filer. The second door was the entry fallback: when refs cannot advance the minute-named cursor, the pass streamed the ref'd file's tail inline, and the client applies inline events unfiltered - the same tail it already applied from chunks. Entry passes for chunk clients now advance the cursor without delivering; everything they skip is covered by the refs already sent or the deltas the next collection ships. * refactor(filer): one gap decision shared by both subscribe loops The post-disk gap tree - guard, drain, resolve, two parks, the re-arm - existed twice, differing only in buffer, watermark space, flush getter, park channels, and reason strings. Four rounds of review fixes have shown the copies drift the moment one is edited alone. gapPass now carries the five differences and the tree lives once; the loops shrink to a three-way switch between reading memory, restarting the pass, and ending the stream. * docs(filer): trim the gap-machinery comments to the why Several blocks had grown to ten-plus lines restating what the tests already pin or retelling one rationale at multiple sites. Keep the non-obvious why - the load-bearing flush ordering, the two timestamp spaces, the refusal-at-equality argument - in a few lines each. * fix(log_buffer): credit an entry's received timestamp to its own window The received-ts capture ran before the rollover check, so an append that sealed the previous window stamped its timestamp onto that window and then lost it in the reset of the new one. The eviction watermark this feeds broke both ways: the sealed window's value was inflated by an entry it does not contain - parking aggregated subscribers on gaps that were drained - and the entry's real window was deflated, proving gaps empty that still held its event on some peer's unflushed path. Credit the timestamp only after the entry lands, when its window is known. * fix(filer): rebase a shipped chunk suffix to logical offset zero A grown file's delta kept the chunks' original file offsets, but the client's chunk reader starts at logical zero and a list opening higher reads as instant EOF - a successfully empty replay, and since chunk clients no longer receive disk entries inline, the appended events were silently dropped. Clone the suffix chunks with offsets rebased to zero; the cut is record-aligned because each append is one uploaded chunk of whole entries, so the suffix decodes as a file of its own. * fix(filer): finish every chunk refs batch with a transition the client acts on Both chunk consumers buffer refs until a non-ref message arrives, so a source with historical logs and a quiet ring - a mount reconnecting after a filer restart is the common case - shipped its backlog and then went silent: the client sat on the refs until the next metadata mutation anywhere in the cluster. The disk step now ends every batch with the empty-notification marker the client already treats as a resume-cursor advance. The same step closes the inline replay: the cursor used to stay at the last file's minute name, so the memory read re-delivered the retained tail of a file the client had just read via chunks - T1..Tn applied twice. The advance-only entry read now runs on every chunk pass, moving the cursor to the true disk content end before memory is consulted. Ordering inside the pass is load-bearing: the entry read can outrun the shipped refs by a chunk appended between collection and read, and the transition timestamp becomes the client's refs filter - stamping it past unshipped content would silently drop that chunk's events on the next delta. The pass therefore re-ships the delta after the entry read, so the transition never exceeds shipped content. Bump-displaced aggregated entries can still arrive inline above the cursor with originals below it; that duplication is bounded and stays within the documented at-least-once residual. * fix(filer): prune ref state at the minute the scan actually stops at The collector compares file names at minute granularity while the prune used the exact-nanosecond scan bound, so for a cursor at 12:31:20 the 12-30 file was still collected but its sent state was already deleted - the next pass reshipped the whole file, re-creating the duplicate-refs class the state exists to prevent. Truncate the bound to the minute the file names live in. * fix(filer): derive the chunk cursor from the shipped refs themselves The advance-only entry read left the three positions that must agree in each other's blind spots. Its snapshot could trail the second delta's, so a chunk appended between them shipped events newer than the cursor and the memory pass sent them again. And it made the filer decode the tail range on every pass, serialized ahead of the client's own reads by the transition marker - re-introducing a slice of the replay work chunk mode exists to offload. Compute the cursor from the shipped set instead: the final entry timestamp of each filer's last shipped chunk, decoded once through the shared chunk cache. Refs coverage, transition marker, and memory start are then the same number by construction - nothing is decoded twice, nothing is dropped, and the per-pass server cost falls to one cached chunk decode per filer. The second delta and the once-per-position refs gate existed to patch the entry read's snapshot races, so both go with it; the range read survives only as a fallback for legacy chunks that do not decode standalone. * fix(filer): keep the chunk-cursor probe inside the shipped snapshot Three holes in the tail probe, all variations of stepping outside what was shipped. A permanently missing chunk failed the stream before the transition marker, so the client discarded its pending refs and reconnected to the same failure forever - blocking all later metadata behind one dead volume, where every other replay path (including the client's own reader) skips such chunks; the probe now walks back to the last readable chunk, and a filer with nothing readable simply contributes no cursor. The legacy fallback re-listed the logs after the refs were collected, so a concurrent append could push the range end over an unshipped chunk and the marker past events the client never received; it now streams the shipped chunk list itself, so no snapshot other than the shipped one is ever consulted. And a file selected before UntilNs can hold entries past it, which the client filters while still adopting the marker as its checkpoint - a later bounded request then skipped them; the marker is clamped to the bound. * fix(filer): make the cursor probe an exact mirror of the client's reader The probe answered from the server's view of the chunks; the marker's correctness depends on the client's. Its backward walk found the last readable chunk, but the client reads forward and stops at the first unreadable one, never resuming within a file - for readable, missing, readable the marker claimed the suffix the client never applied, losing those entries permanently. Keeping only each filer's final file ref discarded the progress of earlier readable files when that file was wholly missing, rewinding the marker to the start cursor. And a torn trailing size prefix - what a crashed writer leaves - failed the probe where the client reads a clean end, blocking the marker forever on data the client accepts. The probe is now shaped like the reader it answers for: per file the readable prefix, per filer the newest file with content, and no condition escapes as an error - understating the marker only re-ships, overstating loses events, and a probe failure must never block the transition the client is waiting on. Each rule is pinned by a test that fails against the previous shape. * fix(filer): judge chunk readability at the volumes, not the decode cache Two ways the probe's answer could drift from what the client experiences. A chunk this server decoded earlier stays warm in the shared cache after its volume dies, so the probe sailed past a chunk the direct-reading client stops at - marker beyond the unread suffix, entries lost. Every chunk now passes a volume lookup before the cache is consulted; the lookup rides the master client's in-memory map, so the probe stays cheap. And a probe stop was treated as harmless understatement, but the delta had already marked the whole ref sent: a transient server-side failure left the cursor stranded behind shipped content for the life of the connection, parking aggregated streams below the watermark for data the client already holds. The pass now rolls back the sent state of every ref above the file that answered the probe, so unreached refs re-ship and re-probe until the cursor gets there. Re-shipped entries at or below the client's checkpoint are filtered client-side, and batches are marker-separated, so a re-shipped file cannot rewind a merge mid-batch. * test(filer): end-to-end subscribe-loop harness and wire-contract tests Every escaped bug across this change's review rounds lived in an interaction the unit tests could not see: the loop state machine, the disk/memory handoff, or the server/client contract. The harness runs the real SubscribeLocalMetadata loop against a real leveldb-backed filer, faking only the volume layer behind the existing test hooks, and asserts the delivered stream itself. Eight scenarios, each pinning a class this change was reviewed for: the headline evicted-unflushed gap parks and then delivers in full; a ring that evicted nothing serves memory promptly; a backlog-to-live handoff with 1ms-adjacent timestamps across every boundary delivers exactly once; a flush-proven gap over vacuumed log files skips to the retained ring including a single-entry window; a bounded subscription terminates at its bound; a permanently wedged flush ends in the give-up skip with the stream still alive; and chunk mode is checked against the real client code - pb.ReadLogFileRefs applied to the shipped refs must cover everything the transition marker claims, with and without a dead volume in the middle. Validated by re-introducing three fixed bugs: the missing eviction guard delivers during the unproven gap, a 2ms cursor error at the handoff drops exactly one event, and resuming at rather than below the earliest retained window loses a single-entry window's sole event - each caught by the scenario built for it. The gap timing knobs become vars so parks run at test speed, a small filer hook swaps the volume-touching read functions, and a sender test pins the refs wire rules the client depends on: never batched, never an envelope, everything in order. * fix(filer): re-ship a partially read answering file, pin the probe's limits The sent-state rollback stopped at files newer than the one that answered the probe. When the answering file itself was only prefix-readable - a dead or transient chunk mid-file - its unread suffix stayed marked sent, and the next append advanced the cursor past it for good. The probe now reports whether the answering file was read through to its end, and a prefix-limited answer re-ships that file too; a torn tail counts as complete, since the client's read ends there as well. The rollback rules live in one predicate with a table test - files below a complete answer stay sent, because the client has moved past them and re-shipping cannot rewind its filter. Two test honesty fixes ride along. The loop harness derived its timestamp base from time.Now() per call, so expectations recomputed across a second boundary drifted by exactly one second; the base is now fixed per harness. And the probe's liveness boundary is pinned as a test instead of a comment: a volume lookup cannot see a dead needle or a stale location inside a resolvable volume, so a warm cache can answer past a chunk the client fails on - accepted because metadata log chunks die volume-at-a-time and the alternative is a real read per probe, which is what the probe exists to avoid. The test states the boundary so changing it is a decision, not an accident. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-02 00:06:22 -07:00
Chris LuandGitHub	e696c2585e	s3 remote: honor s3.support_tagging in UpdateFileMetadata (#10532) ... * s3 remote: honor s3.support_tagging in UpdateFileMetadata The write path already skips tagging when the remote is configured without tagging support, but the metadata-update path sent PutObjectTagging or DeleteObjectTagging unconditionally. On remotes that reject tagging requests, any metadata update failed -- including updates with no tags at all, which land on the DeleteObjectTagging branch. * s3 remote: drop the never-read supportTagging field Every maker set it, nothing read it: the tagging decision is made from conf.S3SupportTagging. Keeping a field that looks like the switch but is not invites exactly the inconsistency the previous commit fixed.	2026-08-01 20:18:59 -07:00
Chris LuandGitHub	0d7173a029	remote storage: actually delete objects when a directory is removed (#10531) ... * remote storage: actually delete objects when a directory is removed On object-store backends RemoveDirectory returned nil without doing anything, so a directory delete synced to the remote as a successful no-op and the objects under that prefix stayed there forever. Nothing surfaced the divergence: the sync logged rmdir, advanced its offset, and the local namespace looked clean. Deleting a bucket-level directory on a filer store that can drop a whole bucket emits no per-child delete events at all, so the single rmdir event was the only chance to clean up the remote. Each backend now lists the prefix and deletes what it finds: S3 in DeleteObjects batches of one listing page, GCS and Azure per object. The prefix always ends with a slash so a sibling like dir2 survives deleting dir, and errors propagate so a failed delete is retried instead of silently skipped. A directory that maps to the bucket root is left alone: wiping every object in the bucket from one namespace event is too destructive, and bucket removal already has its own path. * gcs remote: wrap the per-object delete error The listing error in the same function already wraps, so the delete error should stay inspectable with errors.Is as well. * s3 remote: name the empty-listing test for what it checks The prefix in that test is a normal directory; what is empty is the listing. The bucket-root guard has its own test. * s3 remote: report the scope of a failed delete batch A DeleteObjects response can carry per-key errors for up to a thousand keys. Surfacing only the first hid how much of the batch failed, and surfacing all of them would build an unbounded error string, so report the count with the first failure as the sample.	2026-08-01 20:11:41 -07:00
Chris LuandGitHub	c21d92b70a	test: wait for async write-budget release after pipeline shutdown (#10530) ... Shutdown drops the sealed-chunk map references, but an in-flight uploader goroutine holds the final reference and releases its budget slot only after reacquiring chunksLock. Asserting Used()==0 immediately after Shutdown races those releases on slow runners. Poll with a bounded deadline instead.	2026-08-01 20:04:40 -07:00
Chris LuandGitHub	de00091765	test: random needles always carry at least one byte (#10523) ... A zero-data needle lands in .dat as a size-0 record, byte-identical to a delete marker, so scans that walk .dat count it as deleted. Once in 1024 writes newRandomNeedle produced one, and the idx-head repair then skipped a row TestRepairIdxHeadTombstones_ReadOnlyVolume expected back.	2026-08-01 00:37:22 -07:00
Lisandro PinandGitHub	fa9f471f56	EC scrubbing: list shards for needles failing scrubs in the result output. (#10510) ... This allows to pinpoint failures to a subset of shards, which can then be bisected and potentially reconstructed.	2026-08-01 00:27:06 -07:00
Chris LuandGitHub	c2701955c7	s3: cover three untested STS paths (#10521) ... * s3: test the GetCallerIdentity handler The handler had no test, only XML marshalling, so nothing pinned that a caller presenting session credentials is reported as the assumed role rather than the user who minted the session. * s3: drive AssumeRoleWithWebIdentity over HTTP with a real OIDC token Coverage reached the OIDC path either at the IAMManager service layer or through the Authorization: Bearer shortcut. Nothing exercised the public STS entry point an AWS SDK actually calls, which is where parameter parsing, the IAMManager dispatch and the XML response shape live. * s3: test that every STS route emits an audit entry STS responses go out through WriteXMLResponse, which never calls PostLog, so track() is the only thing that logs them. A route registered outside it would mint credentials with no audit trail and nothing would notice. STS has three routing layers, so a new action is easy to attach to the wrong one. * s3: make the STS tests assert what they claim to cover The audit routing test ran against an uninitialized STS service, so every case answered 503 and a non-404 status was the only evidence the request had reached STS at all - the POST-body case could have been served by the dispatcher's IAM branch and still passed. Back it with a real STS service and assert the STS response namespace, which IAM and S3 responses do not carry. The session policy case checked that the policy travelled in the token rather than that it restricted anything; assert the narrowed bucket is allowed and another bucket is refused. Give the forged-token case the same claim set a valid token gets, so it cannot pass for want of a claim, and cover both rejection paths: a key we do not publish, and a key id absent from the JWKS.	2026-07-31 23:56:04 -07:00
Chris LuandGitHub	ef6a706c0e	master: count only writable volumes as crowded when deciding growth (#10522) ... The crowded map keeps volumes that later became unwritable, so their state survives transient writability flips without flapping. But volumes packed to capacity (fs.mergeVolumes) or turned read-only stay above the crowded threshold and get re-marked on every heartbeat, so the raw map size can permanently exceed the writable count. ShouldGrowVolumes then returns true forever, every assign-path grow request passes the gate, and the periodic grow loop fires too, creating volumes without bound -- worse with -volumePreallocate. Count crowded as the intersection with writables instead: growth checks and the layout gauges only see crowded volumes that can still take writes.	2026-07-31 19:56:45 -07:00
Chris LuandGitHub	1ce106e69d	s3: audit the assumed-role principal and the STS caller (#10519) ... * s3: log the requester's principal ARN in the audit entry An STS session authenticates as an opaque session subject, so requester alone gave an operator no way back to the assumed role or the session name. Record the principal ARN next to the identity name and emit it as requester_arn. * s3: record the caller identity in the STS handlers AssumeRole, GetFederationToken and GetCallerIdentity verify the caller themselves and are not wrapped by the auth middleware that records the identity, so every audit entry for minting a session had an empty requester. * s3: resolve the audit principal ARN the way policy evaluation does A JWT-authenticated identity carries no PrincipalArn — the auth layer hands the principal over in a request header — so reading the field directly left requester_arn empty for OIDC callers. buildPrincipalARN is the resolver the policy path already uses: header first, then the identity's own ARN, then a synthesized user ARN for legacy identities that have none.	2026-07-31 19:51:03 -07:00
Chris LuandGitHub	fa432f9a6a	s3: keep an admin's role session scoped to the role (#10520) ... * s3: keep an admin's role session scoped to the role AssumeRole copied the caller's admin standing into the minted session as the is_admin claim, which short-circuits base policy evaluation. An admin assuming a scoped-down role therefore kept full access and the role's attached policies, explicit denies included, were never evaluated. Only a session the caller assumed for itself carries the claim now — a legacy static admin has no IAM policies for such a session to inherit. * s3: name the caller when it assumes a session for itself An identity that carries no principal ARN left the self-assumed session with an empty role name in its assumed-role ARN. callerPrincipalArn synthesizes the canonical user ARN for that case.	2026-07-31 19:50:59 -07:00
Chris LuandGitHub	d8d29c4ede	s3: carry storage class in the cached listing metadata (#10516) ... A listing on a versioned bucket is served from metadata cached on the .versions directory entry so the whole listing is a single scan. The cache carried size, mtime, ETag, owner and the delete-marker flag but not the storage class, so newListEntry found none and fell back to STANDARD. The result was that HEAD and the listings disagreed about the same object: HEAD reported the class the object was stored with, while ListObjectsV2 and ListObjectVersions reported STANDARD for every object. Clients that filter or tier on storage class act on the listing. Caches the class alongside the other listing fields, clears it with them, and copies it in the routed RECOMPUTE_LATEST path so both finalize paths agree.	2026-07-31 19:48:17 -07:00
Chris LuandGitHub	f582c8451b	s3: report a peer that went away as ClientDisconnected, not IncompleteBody (#10511) ... * s3: report a peer that went away as ClientDisconnected, not IncompleteBody A streaming PUT whose body ends early is always reported as IncompleteBody (400). That collapses two cases with opposite causes: the peer vanished mid-upload, and the peer sent fewer bytes than it promised while still connected. The first points at the network path, the second at the client, and once merged they cannot be told apart from the logs. Split out ClientDisconnected (499) and select it when the request context shows the peer is gone. The upload itself keeps running on a background context so chunks still finish, which means cancellation races the read error; a missed signal degrades to IncompleteBody exactly as before. * s3: note what request-context cancellation is taken to mean	2026-07-31 19:43:45 -07:00
Chris LuandGitHub	7b8188fc41	wdclient: age vid map entries by generation instead of chaining snapshots (#10506) ... * wdclient: age vid map entries by generation instead of chaining snapshots The vid map kept its history as a linked list of past snapshots, trimmed in place by storing nil into a node's cache pointer. That cost up to six full copies of the volume-location map, a recursive walk taking a different lock per level, and deletes that had to cascade through every generation. It also had to special-case explicitly-empty entries, or fallback would resurrect locations a newer snapshot had cleared. Keep one map instead, and stamp each entry with the generation it was learned in. resetVidMap bumps the generation and drops entries that were not relearned within the retained window, which is the same retention the chain provided: an entry survives DefaultVidMapCacheSize resets. The first write of a generation replaces an entry rather than merging into it, so a volume that moved answers with where it is now — the property a fresh map per reset used to give for free. Entries are copy-on-write, so locations handed to a caller are no longer shifted underneath it by a concurrent delete. The map is never swapped now, so the client-side lock and its stable / current accessors go away with it. * wdclient: make vid map entries immutable and drop them once emptied Review follow-up. Updating an entry in place left the copy-on-write guarantee resting on callers never holding the entry pointer; install a new entry instead, so the rule is simply that a stored entry never changes. Deleting a volume's last location now drops the entry rather than keeping an empty one, which a client that never resets would otherwise hold for every volume it ever saw deleted. Lookups already treat an empty entry as a miss, so nothing observable changes. * wdclient: let the newest generation decide between regular and EC locations GetLocations checked the regular map first whatever its generation, so a volume that was EC encoded kept answering with the regular copies the previous master knew until they expired — for as long as the retained window, since nothing relearns a copy that no longer exists. The snapshot chain did not have this problem: the newest map was consulted first and only a volume it knew nothing about fell through to older ones. Restore that by comparing generations, with regular copies winning a tie, since a tie means one generation reported both.	2026-07-31 02:16:33 -07:00
Chris LuandGitHub	ae4839e005	mount: keep a sealed chunk alive until its own upload finishes (#10504) ... Sealing a logic chunk index that already held a sealed chunk dropped the old chunk's only reference and freed its page chunk. That chunk's upload may not have started reading it yet — Execute() returns as soon as the job is handed to a goroutine — so mem.Free could hand a live 2 MiB mem chunk back to the slot pool, the next NewMemChunk would overwrite it, and the in-flight upload shipped whatever bytes were there. Under fio randwrite the volume server rejected those needles with "Content-MD5 did not match md5 of file data" and the FUSE write failed with EIO. Give the sealed chunk a second reference for its upload, dropped only by the upload itself, and let the upload unindex itself only while it still owns the index — the unconditional delete could evict a newer sealed chunk and hide its dirty pages from readers.	2026-07-31 01:16:02 -07:00
Chris LuandGitHub	7fb36025b3	wdclient: read the vid map cache link before the live map (#10505) ... resetVidMap trims the cache chain by storing nil into a node's cache pointer once it ages past vidMapCacheSize. A lookup that missed in its own map and only then loaded that pointer could find the link already severed, reporting "not found" for a volume that stayed resolvable the whole time. Load the link first, while it is still guaranteed live. A published vidMap's cache pointer only ever goes from its ancestor to nil, so reading it earlier can never yield staler history.	2026-07-31 00:52:04 -07:00
Chris LuandGitHub	3514925581	filer: let a nested path rule turn worm off (#10503) ... * filer: let a nested path rule turn worm off mergePathConf ORs the booleans, so worm set on a bucket could never be lifted on a directory under it, while every string field is overridden by the more specific rule. Make worm tri-state instead: unset inherits, set wins. readOnly, fsync and disableChunkDeletion keep the OR, so a nested rule still cannot escape a lock the bucket set. Configurations written before this carry an explicit "worm": false on every rule, because they are marshalled with EmitUnpopulated. Reading those back as an override would quietly drop worm from nested paths, so filer.conf is now stamped with a version and the flag is dropped to unset when the version predates it. * filer: copy the worm value out of the matched rule mergePathConf aliased the pointer into the merged result, so a caller that wrote through it would reach into the stored rule.	2026-07-31 00:34:22 -07:00
Chris LuandGitHub	4dc1b70b2f	test: pin that a .vif replication outranks the superblock (#10499) ... * test: pin that a .vif replication outranks the superblock Store.ConfigureVolume rewrites the .vif and never the replica-placement byte in the .dat, so that byte keeps whatever the volume was created with for good. readSuperBlock reads it and then overrides it from the .vif, which is what makes a replication change take effect and survive a remount. Invert that and every replication change silently reverts on the next mount, while the .vif on disk still records what the operator asked for -- a durability setting quietly going back to its old value, with nothing to indicate it. Worth pinning rather than reading off the code, because the field beside it resolves the other way: version takes the superblock over the .vif. Two fields, one function, opposite precedence, each a line to invert wrongly. Covers the empty case too, since a .vif that declares no replication has to leave the superblock standing or a volume whose replication was never configured would be forced to whatever the zero value parses as. * test: drop the unreachable nil check on MaybeLoadVolumeInfo It initialises the returned pointer before the existence check and every return is naked, so it never yields nil. Guarding against it implied a contract the callee does not have.	2026-07-30 17:03:38 -07:00
Chris LuandGitHub	63d5140485	s3: allow copying an object onto itself in a versioned bucket (#10497) ... * s3: allow copying an object onto itself in a versioned bucket The copy writes a new version instead of overwriting in place, which is how an earlier version is restored. Buckets with versioning off or suspended keep rejecting a self-copy that changes nothing. * s3: cover the suspended-versioning self-copy rejection Suspended versioning overwrites the null version in place, so a self-copy that changes nothing stays rejected. Pin that alongside the never-versioned case.	2026-07-30 13:00:41 -07:00
Chris LuandGitHub	e7a678fa72	s3: keep the list marker exclusive for versioned objects (#10496) ... * s3: keep the list marker exclusive for versioned objects A versioned object lives in a "<key>.versions" directory, so the entry name never matched the marker and start-after/marker returned the marker key itself. * s3: match the list marker against the raw entry name too A backend that echoes the marker it was given returns the ".versions" directory name, which no longer matched once the comparison used the object name alone. Cover both, and unit test each half.	2026-07-30 12:57:31 -07:00
Chris LuandGitHub	ccf5dc34e9	test: stop comparing two JWTs minted a second apart (#10495) ... TestProxyReadDropsCallerJwtQueryParam mints a read token up front and requires the token the volume server would evaluate to equal it byte for byte. The expiry claim has one-second resolution -- GenJwtForVolumeServer sets it from jwt.NewNumericDate(time.Now().Add(...)) -- so two mints on either side of a tick produce different strings for the same authority and the same file, and the assertion fails for a reason the test is not about. It surfaces on the 32-bit job, where the runner is slow enough that the HEAD subtest (the second one, after a full proxy round trip) lands in a later second than the mint at the top of the test. Confirmed directly: minting the same file id with the same key either side of a boundary yields different tokens. Assert what the test is actually about instead -- that the credential decodes against the read key and authorizes this file id -- which holds whatever second it is minted in, and is a closer statement of the property than string equality.	2026-07-30 12:23:12 -07:00
Chris LuandGitHub	78ed665557	webdav: answer PROPFIND child stats from the listing (#10492) ... * webdav: answer PROPFIND child stats from the listing golang.org/x/net/webdav discards the FileInfo that Readdir returned and stats every child again, five times over, so a PROPFIND on a directory costs five sequential filer lookups per entry: 15006 lookups and 1.4s for 3000 subdirectories, 24s for 60000. Windows Explorer times out well before that. Hold the entries a listing already fetched for the lifetime of the request and serve those stats from them - 6 lookups and 0.03s for the same 3000 entries. WebDavFile.Stat has to stop dropping its request context for the held entries to be reachable. * webdav: keep the request context on the lookups a listing drives stat, Readdir and Seek reached the filer on context.Background(), so a client that walked away left the listing streaming and the lookups running. Seek also missed the entries the listing had already fetched. Write and cleanup paths keep their own context - a cancelled request must not abandon a flush half done.	2026-07-30 11:58:14 -07:00
Chris LuandGitHub	46ceb253b0	telemetry: report anonymous cluster stats by default (#10488) ... The reports are what tell us which versions and cluster sizes are actually in use, and almost nobody flips the flag on, so the numbers we have are close to useless. Default it on for master, server and mini, and say in the flag help and the startup log how to turn it off. Nothing new is collected: still an in-memory cluster id that changes on restart, version, os, server counts, volume count and disk bytes, sent once a day by the leader master only.	2026-07-29 15:11:00 -07:00
Chris LuandGitHub	c2183566b6	ec.encode: name the shard ids an aborted deletion found (#10486) ... Counting by node lost the per-node ShardsInfo, and with it the shard ids the old summary printed -- the message an operator gets when the pre-delete check refuses now says only how many shards each node holds. That is the wrong half. A set holding shards 0-9 and one holding 4-13 are both "10 shards", and only the ids say whether what survived can rebuild the volume, or which node to go looking at. Keep the count and list the ids beside it.	2026-07-29 14:21:55 -07:00
Chris LuandGitHub	c4798979d8	ec.encode: count shards wherever they landed before deleting the source (#10483) ... generateEcShards writes shards beside the source volume, so encoding a volume that lives on a non-default medium puts them on that medium while -diskType still says hdd. The pre-delete check counted only the -diskType bucket, so it saw a complete set as zero shards, called it unrecoverable and aborted -- leaving the volume as both a .dat and a full shard set, which every later reader then disagrees about. Count by node across disks, as waitForEcShardsToRegister in the same file already does. The spread check is unaffected: it locates shards through collectEcShardBitsByNode and only uses diskType to find free slots.	2026-07-29 13:51:46 -07:00
Chris LuandGitHub	167c114dae	ci: fix FUSE mounts against the new runner image (#10484) ... * ci: restore the setuid bit on a shadowed fusermount3 Newer ubuntu-22.04 runner images carry a source-built fusermount3 in /usr/local/bin that shadows the distro one in PATH and is not setuid root. go-fuse looks the helper up through PATH, so every unprivileged mount fails with "mount failed: Operation not permitted". * test: fail a fuse test as soon as its mount process dies A mount that cannot mount at all exits within a second, but the harness still waited out the 30s readiness timeout and then reported "mount point not ready within timeout", leaving the real cause buried in the log tail. Watch the child processes and report their exit instead. * mount: report a failed mount without a goroutine dump A mount failure is an environment problem - no /dev/fuse, fusermount not setuid, stale mount point - and the all-goroutine stack dump Fatalf adds buries the one line that says so.	2026-07-29 13:32:35 -07:00
Chris LuandGitHub	4149346bb7	s3: register the advertised ip with the master (#10482) ... * s3: register the advertised ip with the master The cluster address came from the bind ip, falling back to the auto-detected interface, so -ip never reached the S3 registration. weed mini -ip=localhost binds the wildcard and ended up registering whatever interface happened to sort first -- on a host with VPN interfaces, an address that stops routing once the tunnel drops. IAM changes are pushed to registered S3 servers over gRPC, so every mutation then blocked the full 10s propagation deadline before logging a failure, and cluster.ps and the admin UI listed a node nothing could reach. Identities still arrived through the /etc/iam metadata subscription, so this cost latency and visibility, not credentials. Add an advertise ip to the gateway option, preferring it over the bind address, and wire the parent -ip through server, filer and mini. * s3: treat any unspecified bind address as a wildcard net.ParseIP + IsUnspecified covers ::, [::] and the expanded IPv6 forms instead of only the 0.0.0.0 literal, so an IPv6 wildcard bind no longer registers an address peers cannot dial. Host names parse as nil and stay addresses in their own right. Apply the same guard to the advertised ip.	2026-07-29 10:30:46 -07:00
Chris LuandGitHub	0002e5cc7f	s3api: load document-style policies from the advanced IAM config (#10481) ... * s3api: load document-style policies from the advanced IAM config The advanced IAM file doubles as the S3 identity config when only -s3.iam.config is given. protojson drops its "document" field, so every policy landed with empty content and warned "skipping invalid policy" on each reload. Worse, if the same file also declares identities the empty content sticks in the policy map and fails the whole runtime policy sync into the IAM manager, so policies created later never reach it. * iam: skip an unparsable policy instead of failing the whole runtime sync One policy the engine cannot parse aborted SyncRuntimePolicies before it touched anything, so every other policy stayed unsynced and the engine kept serving whatever it last held. * s3api: reject a non-role RoleArn in AssumeRole as a bad request arn:aws:iam:::user/name can never resolve to a role, but the handler ran it through the trust-policy check and answered "not authorized to assume role", pointing the caller at a permission problem they do not have. * s3api: build the policy content before touching the entry Deleting "document" up front meant a marshal failure left the policy with neither field, so a later rewrite would emit it with no definition at all. * iam: pin the fail-closed handling of an unparsable policy Say in the comment that dropping it from the desired set deletes it from the engine on purpose, and cover it with a test. * s3api: widen the non-role RoleArn test to canonical ARN shapes The reported ARN omits the account id; a user ARN that carries one, and a non-principal ARN, must be rejected the same way.	2026-07-29 10:30:30 -07:00
Chris LuandGitHub	13176b4edd	volume: recover .idx rows overwritten by tiered deletes (#10474) ... * volume: recover .idx rows overwritten by tiered deletes A delete on a read-only volume backed by a remote tier used to write its tombstone row at .idx offset 0 rather than appending it, so each delete overwrote one more row at the front and lost the Put rows indexing the first needles in .dat. Those needles 404 even though .dat still holds them, and rebuilding .idx with weed fix means stopping the server and pulling the whole .dat back from the tier. The damage has a fingerprint -- .idx opening with a run of offset-0 tombstones, which a healthy .idx never does -- and .idx and .dat grow in lockstep, so the lost rows indexed exactly the first N .dat records. Detect it at load and re-derive them from a header-only walk over the head of .dat, cheap even against a remote tier, appending only the keys the .idx no longer names. * rust volume: mirror the .idx head tombstone recovery Port the Go detection and repair: an .idx opening with a run of offset-0 tombstones lost the Put rows indexing the first needles in .dat, so re-derive them at load from a header-only walk over the head of .dat and append the keys the .idx no longer names. * volume: put recovered .idx rows back in front instead of appending Appending left the offset-0 tombstone run at the head, so every later load re-walked .idx to the tail to notice the volume was already recovered, and the rows for the head of .dat sat past the .dat-tail row -- costing CheckVolumeDataIntegrity its O(1) path and breaking the ascending append order BinarySearchByAppendAtNs assumes. Rewrite .idx as the recovered rows followed by its current contents, through a temp file and a rename. .idx is back in .dat append order, so a later load stops after reading one row. * volume: keep the .idx mode when the repair replaces it The recovery renames a fresh temp file over .idx, so a fixed 0644 (Go) or whatever the umask allows (Rust) would silently widen an index an operator had locked down. Carry the mode off the file being replaced.	2026-07-28 16:48:30 -07:00
Chris LuandGitHub	4b0d09683a	iceberg: read manifest lists that omit the Avro format version (#10475) ... * s3tables: read Iceberg manifest lists that omit the Avro format version The Iceberg spec pins the Avro header metadata of manifest files but says nothing about manifest lists, so writers disagree. Java and PyIceberg record "format-version"; DuckDB writes no header metadata at all. iceberg-go reads a missing entry as v1, so every v2 manifest listed in a DuckDB-written list is rejected with manifest file's 'format-version' metadata indicates version 2, but entry from manifest list indicates version 1 and, because v1 has no "content" field, delete manifests silently decode as data manifests. ReadManifestList derives the version from the record schema the writer embedded - v2 added "content" and the sequence numbers, v3 added "first_row_id" - and splices it into the header before handing the bytes to iceberg-go. Lists that already carry the entry, and input that is not a parseable Avro container, go through untouched. * iceberg: parse DuckDB-written manifest lists in maintenance and data preview Every manifest list read - the four maintenance operations and the admin table data preview - went straight to iceberg-go, so tables written by DuckDB failed detection and all of compact, remove_orphans, rewrite_manifests and expire_snapshots before they touched anything. Route them through s3tables.ReadManifestList, which recovers the format version the writer left out of the Avro header. This also restores the manifest content type on those tables: with the list read as v1 every delete manifest looked like a data manifest, which hid deletes from the compaction guard and made the preview report a table with position deletes as having none.	2026-07-28 16:42:17 -07:00
Chris LuandGitHub	ac6f3c92ef	s3api/iceberg: report the reason a table schema was rejected (#10473) ... * s3api/iceberg: report the reason a table schema was rejected newTableMetadata swallowed the iceberg-go error and returned nil, so every schema the metadata builder refused came back as a bare 500 "Failed to build table metadata". A v3-only column type is the common case: creating a table with a variant field but no format-version 3 property leaves the client with nothing, while "variant is not supported until v3" sits in the server log. Return the error instead and classify it. Schema, spec and argument failures are the caller's input, so they answer 400 with the underlying reason; the rest stay 500. Paths that build placeholder metadata with no schema keep their existing 500 via newEmptyTableMetadata. * s3api/iceberg: fail LoadTable when placeholder metadata cannot be built buildLoadTableResult dropped a nil from the placeholder path straight into the response. That serializes as "metadata":null under HTTP 200, which no Iceberg client can parse -- a worse outcome than the 500 the nil was meant to signal. Return an error instead and let the five callers answer 500. The nil-return convention goes away with it, so the commit and transaction paths check an error rather than a sentinel. * s3api/iceberg: route rejected schemas through writeManagerError The two helpers added here duplicated work the package already does. writeManagerError is the canonical error-to-response mapper -- it already downgrades client-input failures to 400 and defaults the rest to 500 -- so teach it the iceberg-go schema and spec sentinels instead of standing up a parallel classifier. The placeholder wrapper was a pure alias for newTableMetadata with nil arguments; call that directly. No behavior change beyond the 500 message, which now reads err.Error() like every other manager error rather than carrying its own prefix.	2026-07-28 14:44:56 -07:00
Chris LuandGitHub	9351202ca9	volume: scan for on-disk EC shards when staging a decoded volume (#10465) ... The staged-new-volume placement skipped a disk holding the vid's EC shards using only the in-memory ecVolumes map, missing a shard present on disk but not mounted. Also scan the candidate disk for <vid>.ecNN files, so the promise holds regardless of mount state. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu	2026-07-27 19:37:16 -07:00
Chris LuandGitHub	3b3e8af430	volume: skip a shard-holding disk when staging a decoded volume (Go+Rust) (#10464) ... volume: skip a shard-holding disk when staging a decoded volume ReceiveFile staged-new-volume mode picked any free disk of the target medium. Skip a disk that already holds the vid's EC shards (Go DiskLocation.FindEcVolume / Rust ec_volumes), so a decoded .dat never lands in the same directory as a shard. This lets a caller safely stage onto a shard host that has a spare disk, instead of requiring a host with no shard of the vid at all. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu	2026-07-27 18:40:57 -07:00
Chris LuandGitHub	9c37e52c9b	volume: EC decode onto a clean peer via staged-new-volume adopt (Go+Rust) (#10463) ... Decoding EC shards back to a normal volume in place reconstructs <vid>.dat in the shards' own directory, so the vid is momentarily registered as both an EC and a normal volume in one location — the load/scan path then sees it as both, risking mount ambiguity and needle loss. VolumeEcShardsToVolume still supports that in-place path; this adds the primitives to decode onto a *clean* peer instead: - ReceiveFile gains a staged-new-volume mode: when the volume does not exist here and ReceiveFileInfo.disk_type is set, pick a free-slot disk of that medium and write <base><ext>.copying (not a valid volume name, so the scanner never half-loads a partial push). - VolumeEcShardsToVolume gains from_staged: adopt the pushed .dat/.idx/ .vif — rename .copying into place under a .note in-progress marker, then mount — so <vid> lands on the peer only as a normal volume. The caller decodes the shards off-box and streams the finished volume to a peer holding no shard of the vid on the target medium. Go and Rust volume servers get identical handlers. Proto: ReceiveFileInfo.disk_type (12; 8-11 reserved for versioned-EC), VolumeEcShardsToVolumeRequest.from_staged (3) + disk_type (4). Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu	2026-07-27 17:56:17 -07:00
Lars LehtonenandGitHub	2bea4dd610	chore(weed/s3api): prune dead code (#10462)	2026-07-27 17:48:20 -07:00
Chris LuandGitHub	62c4333074	s3: list the buckets an attached IAM policy grants (#10458) ... * s3: list the buckets an attached IAM policy grants ListBuckets served an identity authorized by an attached IAM policy only the buckets it had created itself. A user granted s3:ListBucket on a bucket someone else provisioned could GetObject and ListObjectsV2 against it, but the bucket never showed up in the listing any S3 client uses to build its bucket picker. The owner-index fast path is only valid for an identity whose grants name every bucket it can reach, and the routing check assumed a policy could never be enumerated. Read the names out of the policy instead: statements that allow s3:ListBucket on a concrete bucket ARN become candidates, and the per-bucket permission re-check still decides what is listed. A policy that can reach a bucket it does not name -- a wildcard resource, a policy variable, a NotResource, an STS session policy -- falls back to the full scan, which evaluates the policy per bucket. * s3: share the attached policy name lookup authorizeWithIAM and the ListBuckets enumeration both built an identity's policy names the same way, its own plus the ones from its enabled groups. Pull that into one helper so group eligibility is decided in a single place. * s3: read policy actions the way the IAM authorizer matches them The IAM authorizer matches action names case-insensitively, so a policy granting "S3:LISTBUCKET" or "S3:*" authorizes a list. The ListBuckets classifier read those actions with the local case-sensitive matcher and found no grant, so once the owner index was ready the buckets that policy allows dropped out of the listing. Match the action the looser way in the classifier: case-insensitive, and true for any pattern holding a policy variable. Over-matching only costs a candidate the per-bucket permission check then rejects, while under-matching hides a bucket the caller can read. * s3: infer a multipart grant in any case The classifier matches action patterns case-insensitively but looked the requested action up in a canonical-cased set, so "S3:UPLOADPART" missed the s3:PutObject inference that the authorizer makes. Key the set for lookup in lower case, matching how the IAM authorizer holds it.	2026-07-27 16:42:01 -07:00
Chris LuandGitHub	5536d88fbb	azure: let the blob endpoint be configured (#10460) ... * azure: let the blob endpoint be configured The service url was always derived as <account>.blob.core.windows.net, which leaves out Azure Government, Azure China, and private endpoints. Name the blob service url instead and those accounts become reachable. The url has to be https, since the account key or the bearer token would otherwise travel in the clear. * azure: reject an endpoint that carries no hostname A url like https://:443/ has a host of ":443", so the emptiness check on Host let it through and the request only failed once it reached Azure. The hostname is what has to be there.	2026-07-27 16:41:13 -07:00
Chris LuandGitHub	fee3fcb55a	mount: report data sizes to df with -df.logical (#10459) ... df on a mount shows the space the cluster gives up to the data: every replica of a regular volume, every shard of an ec one. That is the honest answer for capacity planning, but it is not the question a user asks when they want to know how much of their data is stored. Add -df.logical. The master reports the logical sizes alongside the raw ones: one replica per regular volume, the data shards of each ec volume counted once. Free space is divided by the copies the requested replication makes, so used plus available stays the amount of data the mount can still write, and it comes off the cluster-wide usage rather than one collection's, since capacity is cluster-wide too. Statistics through a filer resolves an unset replication to the filer's default rather than the master's, matching where the writes it is sizing for actually land. The flag governs the quota check too, so a mount has one notion of how much it is using. A filer that predates the new fields sends zeros, and the mount keeps reporting the raw sizes.	2026-07-27 14:28:29 -07:00
Chris LuandGitHub	152f1a2096	master: count EC volumes in statistics used size (#10457) ... Statistics aggregates the volume layouts of a collection, but EC volumes are tracked outside collectionMap, so they were reported as nothing. A mount over a cluster whose volumes have mostly been encoded showed a df used size of a few GiB against terabytes of EC data. Walk the data nodes and add the EC volumes of the requested collection. Every shard copy counts, parity included, the way a regular volume's used size counts every replica, so used size stays the space the cluster actually occupies. File count comes from the volume-wide .ecx and .ecj counts, taking the largest a holder reports rather than summing them: both files travel with the shards on a move, so several nodes can report the same tombstones.	2026-07-27 14:24:51 -07:00
Chris LuandGitHub	3ae4e9c563	azure: authenticate with Entra ID instead of a storage account key (#10456) ... * azure: authenticate the blob sink with Entra ID Shared account keys have to be distributed and rotated everywhere a sink runs. Leaving account_key empty now falls back to the identity chain, so a workload identity or managed identity carries the authorization instead. * azure: authenticate remote storage with Entra ID The remote storage client demanded an account key and refused to start without one. Fall back to the identity chain when it is absent, and let azure.client_id pin a user-assigned identity. * azure: reject a malformed storage account name The account name is interpolated into the service URL, so a name carrying a "/", "?" or "@" moves the authority elsewhere and an authenticated request follows it. Hold callers to Azure's own naming rule instead. * azure: keep a leftover environment key off the identity path A configured client id asks for Entra ID, but AZURE_STORAGE_ACCESS_KEY still filled in the account key behind it. An old mounted secret would go on authenticating until it rotated, and the failure then blamed the key. * azure: say what the identity path reads from the environment A pinned client id alone is not enough for workload identity: the tenant and the projected token come from the environment, and missing them only surfaces later, when a token is first requested.	2026-07-27 14:12:14 -07:00
Chris LuandGitHub	6b6e6d8547	s3: apply filer identity changes despite a static config file (#10392) ... * s3: apply filer identity changes despite a static config file A -config file with inline identities disabled the metadata-subscription reload entirely, leaving the best-effort filer->s3 push as the only way s3.configure changes could reach a running gateway. Reload on IAM events regardless: the merge keeps the file's identities protected, and a full credential-manager snapshot now also drops dynamic identities the store no longer has, so revocation works without a restart. * s3: log identity propagation failures as warnings * s3: retry failed IAM reloads and reconcile policies and groups An event-driven reload that fails now hands off to a coalescing retry loop, so a transient filer error cannot strand a revoked credential until the next IAM event. Full-state merges also drop dynamic policies the store no longer has, keeping the static file's, and treat the group snapshot as authoritative even when empty. * s3: serialize IAM configuration loads The SIGHUP file reload, subscription reloads, the retry loop, and the postgres poll run on different goroutines. Without an end-to-end lock a load holding an older store snapshot can commit after a newer one and revert it. Hold reloadMu from snapshot through commit in both load entry points; partial merges from pushed updates stay lock-free and self-heal through the next event-driven reload. * s3: keep static-file groups through full-state reconciliation Group names from the static config file are tracked like identities and policies, and a full snapshot that does not carry them keeps the current definition and its memberships instead of dropping them. * credential: include groups in postgres configuration snapshots Full-state reconciliation treats absent groups as deleted, so a snapshot that never carries them would erase every dynamic group. * s3: revoke static-file groups dropped from the config file A file reload is authoritative for the file's group set while keeping dynamic groups, mirroring how full snapshots are authoritative for dynamic groups while keeping the file's. * credential: fail filer snapshots on unreadable entries A skipped identity or policy file made the load report success with an incomplete snapshot, which reconciliation reads as deletion and the retry loop never sees. Unparseable content is still skipped: it is durable, matches boot behavior, and must not block reloads forever. * s3: ignore groups in static config files Groups are managed through the IAM API and the dynamic store; no deployment defines them in a bootstrap config file. Ignoring them with a warning removes the two-directional group merge: full snapshots are plainly authoritative and file reloads never touch groups.	2026-07-27 14:06:04 -07:00
Chris LuandGitHub	a7f4b88a61	s3: require a bucket-policy action to write a bucket policy (#10444) ... * s3: require a bucket-policy action to write a bucket policy PutBucketPolicy and DeleteBucketPolicy were gated on ACTION_WRITE, the same action that grants object writes. An explicit Allow in a bucket policy short-circuits IAM entirely -- authRequestWithAuthType sets policyAllows and skips VerifyActionPermission -- so anyone who could write an object could author a policy granting itself, or anonymous, anything on the bucket. That is what separates a bucket policy from the sibling bucket controls also gated on ACTION_WRITE: rewriting cors or lifecycle can destroy data, but only a policy hands out access. Give the two verbs their own actions, mapped to the AWS names that were already defined but unrouted. ACTION_ADMIN would also have closed it, but it resolves to s3:* for IAM identities, forcing a blanket grant on a user holding a precise s3:PutBucketPolicy. Admins are unaffected, since isAdmin short-circuits CanDo, and an operator can delegate with PutBucketPolicy:bucket. The route binding is asserted from the router source: checking the action constants alone still passes when the route says ACTION_WRITE. * s3: also read the action from a direct iam.Auth call in the route test Routes read iam.Auth(cb.Limit(handler, ACTION)), a multi-value pass-through: Limit returns (http.HandlerFunc, Action) and those become Auth's parameters, so the action Auth authorizes on is Limit's second argument and the two cannot disagree -- Auth(Limit(h, X), Y) does not compile. A route that skipped Limit and called Auth with its own action would compile, though, and the test reported that as a missing route rather than as the wrong action. Recognise the two-argument Auth form so it names the action instead. * s3: make the bucket-policy actions grantable through an IAM policy The new actions close the escalation only if an operator can grant them, and they were not reachable: MapToStatementAction had no entry for PutBucketPolicy, so an IAM policy naming s3:PutBucketPolicy was rejected outright with "not a valid action". GetBucketPolicy was unmapped the same way. DeleteBucketPolicy was mapped, but to ACTION_ADMIN -- granting an identity permission to delete a bucket policy handed it full administrative access. Map all three to the actions the router now uses, and add the reverse direction so an identity holding them renders back as a policy statement instead of a bare "s3:". * admin: offer the bucket-policy permissions in the user editor The two new actions are otherwise only grantable by hand-editing identity JSON or by calling the IAM API, so an operator using the UI cannot delegate bucket policy management without granting Admin. Regenerating this file also picks up codegen the repo has not taken yet: the checked-in _templ.go files were produced by templ v0.3.1001 while go.mod pins v0.3.1020, so the generator rewrites the attribute-value calls. That churn is confined to this one file; running `make generate` in weed/admin reproduces it across all 36.	2026-07-26 00:52:22 -07:00
Chris LuandGitHub	c7d0477117	volume: widen the gRPC admin gate and stop it drifting (#10443) ... * volume: gate the admin RPCs that only shell and workers call checkGrpcAdminAuth covered 19 of the 48 VolumeServer RPCs, so an operator who sets -whiteList expecting it to cover the gRPC surface gets partial coverage. Extend it to ten that mutate state and are only ever called by the shell or a worker: SetState, VolumeCopy, the EC generate/rebuild/copy/unmount/to-volume pair, both tier moves, and VolumeTailReceiver. That is safe because the same callers already reach gated RPCs today -- VolumeMarkReadonly, VacuumVolume*, VolumeEcShardsDelete, VolumeDelete -- so a whitelist deployment already lists those hosts. Nothing here is on a master or peer path, which is what made the earlier fail-closed gate break multi-host clusters. The split is by caller rather than by blast radius: the guard matches a peer IP against the whitelist, and a whitelist holds masters, shell hosts and workers, not every peer volume server. Gating a call one volume server makes to another would break replication, EC and tiering, so those stay open. Two test fakes embedded a nil grpc.ServerStream and only implemented Send; they now implement Context, which the streaming RPCs read to authorize. * volume: fail the build when a gRPC method skips the admin gate The admin gate is an opt-in list in a 48-method service, which is how it drifted down to covering 19 of them: nothing tied adding an RPC to deciding whether it needed the gate. Parse volume_server.proto, walk the AST of every *VolumeServer method, and require each RPC to either call checkGrpcAdminAuth or appear in ungatedVolumeServerRPCs with the reason it stays open. A stale entry naming an RPC that no longer exists fails too, so the list can't quietly stop exempting anything. The exemptions are the cluster-internal calls -- replica sync, EC shard distribution, vacuum reads, backup, tailing -- plus the read-only and liveness RPCs. Closing the cluster-internal ones needs a peer identity rather than an IP whitelist; recording them here makes that a visible decision instead of an omission. The AST walk also corrects the count: a line-window scan credits VacuumVolumeCheck and VolumeServerStatus with a neighbouring function's guard.	2026-07-25 23:53:29 -07:00
Chris LuandGitHub	be81b9d5d7	volume: fix EC decode/reconstruct index locality under -dir.idx (#10442) ... * volume: fix EC decode/reconstruct index locality under -dir.idx EC->replicated decode failed under -dir.idx and on multi-disk with "volume not found on disk". The reconstruct rebuilds the .dat on the data disk but the on-demand VolumeMount scans only the data directory, matching on .idx/.vif; with the rebuilt .idx off in the index directory it matched the volume's leftover EC .vif and skipped the volume as EC metadata. - Resolve the EC .ecx local-first: prefer the copy co-located with the shards over the shared -dir.idx copy, with a non-empty preference so a 0-byte local stub still yields to a valid sibling (the cross-disk fallback). - Co-locate the rebuilt .idx with the .dat at the end of the reconstruct so the mount finds it; sweep .ecx/.ecj from both the data and index directories on Destroy so a stale copy cannot re-mount as a phantom EC volume. - Add VolumeConsolidateIndex: once the EC shards are deleted, unmount, move the .idx/.sdx from the data disk back to the -dir.idx directory (copy fallback across filesystems), and remount. A no-op without -dir.idx. * volume: tests for EC index locality (local-first .ecx, sweep, consolidate) - NewEcVolume prefers a non-empty local .ecx over the shared index dir, and a 0-byte local stub yields to a non-empty shared copy (the #9212 fallback). - Destroy sweeps .ecx/.ecj from both the data and index directories. - ConsolidateVolumeIndex moves a co-located index back to the -dir.idx dir and keeps the volume mounted; no-op without a separate index dir. - RenameOrCopyFile moves a file and drops the source. * volume: relocate the decoded index in place, without a read gap ConsolidateVolumeIndex previously unmounted the volume, moved the index, and remounted it. Between the EC-shard delete and the remount the volume had neither a normal nor an EC form mounted, so a read landing in that window got a not-found (or was proxied away). Move the index in place instead: RelocateIndexTo takes the data-file write lock, closes the needle map and data backend, moves the .idx (and derived .sdx), then retargets dirIdx and reloads — the same close-swap-load CommitCompact uses. The volume never leaves the mounted set, so a concurrent read blocks briefly on the lock rather than failing. The test now writes a needle before consolidating and reads it back after, proving the in-place reload keeps the volume serving. * volume: address review — maintenance guard, no orphan on copy failure - VolumeConsolidateIndex now rejects the request under maintenance mode, like VolumeConfigure and the other mutating volume RPCs. - RenameOrCopyFile rolls the cross-device copy back when the source cannot be removed, so a failed move never leaves two divergent copies (the loader would keep the data-dir one while the idx-dir orphan goes stale). - RelocateIndexTo logs a failed reopen-after-failed-move instead of swallowing it, since that leaves the volume unusable until the next load.	2026-07-25 23:45:02 -07:00
Chris LuandGitHub	83f754763e	filer: make the redis connection settings configurable (#10441) ... The sentinel stores hardcoded a 30s read timeout and a 1m retry backoff. After a sentinel failover every request that picked a pooled connection to the old master sat there for 30s before the connection was retired, and the pool timeout derived from it (read timeout + 1s) queued the rest behind them. The other redis stores took the go-redis defaults with no way to tune anything. Read the dial, timeout and pool knobs from each redis store section instead, keeping the go-redis default for every key left unset.	2026-07-25 19:55:39 -07:00
Chris LuandGitHub	5cac980b32	filer: drop a caller's jwt query param on a proxied read (#10440) ... security.GetJwt reads the "jwt" query parameter before the Authorization header, and the proxy forwarded the caller's whole query apart from proxyChunkId. So on a read, where the filer mints a volume token and sets the header itself, a caller-supplied ?jwt= silently outranked it: the volume server validated a credential the caller chose rather than the one the filer attached, and the read failed with a 401 the filer could not explain. Drop it on the read path, where the filer owns the credential. Writes keep theirs -- the proxy forwards a writer's own AssignVolume token either way, so the query parameter is just a second channel for the same credential and stripping it would break a caller that presents it that way. Nothing in the tree passes a jwt by query; maybeAddAuth always sets the header.	2026-07-25 19:54:41 -07:00
Chris LuandGitHub	6824619c16	s3: chunk uploads at the filer's maxMB (#10439) ... The S3 write path cut fixed 8MB chunks, so an object stored through S3 chunked differently from the same bytes stored through the filer, WebDAV or a mount, and -maxMB had no effect on it. Read maxMB from the filer configuration at startup and use it, falling back to 8MB when the filer reports none.	2026-07-25 11:30:02 -07:00
Chris LuandGitHub	0f718f8509	filer: add a placement overlay seam for the write path (#10437) ... * filer: add a placement overlay seam for the write path New volumes take their disk type, replication, and data center from the explicit request or the matched filer.conf rule. That leaves no way for a feature to steer a whole collection onto a medium without an operator writing an fs.configure rule by hand. Add a generic PlacementOverlay hook on the filer: a func that maps a collection to a placement override, installed by a factory the way the plugin-worker handlers register. detectStorageOption consults it between the explicit request value and the filer.conf rule, so it overrides the rule but yields to a value the caller asked for. The seam names no feature concepts, so it stays generic; a downstream build registers the overlay it wants (e.g. a storage-class Landing tier). Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu * filer: address review on the placement overlay seam Honor ResolvePlacement's ok flag explicitly rather than relying on empty values falling through the util.Nvl chain, and log at V(4) when the overlay steers a collection. Document that RegisterPlacementOverlay is init-only, so the unsynchronized read in NewFiler cannot race the write. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu	2026-07-25 11:20:41 -07:00