PutObjectAcl had four authorization and ownership bugs:
- The handler embedded the resource path into the action
(WriteAcp:bucket/object), and authRequest/CanDo then scoped it to
the request's bucket/object again. A bucket-wide WriteAcp:bucket
grant could never match, so legitimate owners got 403.
- After authRequest succeeded via an IAM or bucket policy, a leftover
identity.CanDo gate re-checked only the legacy Actions list, denying
identities authorized purely by policies.
- For canned and default ACLs, ExtractAcl generated the FULL_CONTROL
grant for the requesting account instead of the object owner. An
admin setting private/public-read on another account's object left
the owner metadata intact but reassigned full control to the admin.
- Objects without stored owner metadata (e.g. written via the filer
outside S3) fell back to treating the requester as the owner, so any
user with a WriteAcp grant could take them over. Non-admins are now
denied; admins keep the takeover fallback.
Grantee validation now also accepts the object's stored owner even when
that account has been removed from the registry, so canned/XML ACLs for
retired owners keep working.
* filer: stop isSameChunks from sorting caller-owned chunk slices
slices.SortFunc reorders the input in place. filer.remote.sync calls IsSameData on a metadata event's NewEntry inside isMetadataOnlyUpdate and later stamps the filer entry under an IF_ENTRY_EQUAL precondition carrying that same entry. The ETag-sorted chunk list never matches the stored entry, so every stamp of a multi-chunk object fails, synced_mtime_ns stays zero, and dirty objects are re-uploaded forever. Sort clones of the slices instead.
* filer: test IsSameData leaves input chunk order unchanged
Guards the clone-then-sort fix: a regression back to in-place sorting would reorder caller-owned chunk slices and reintroduce the remote-sync IF_ENTRY_EQUAL mismatch.
* s3api: copy the trailer checksum before reading the next trailer line
parseChunkChecksum kept the checksum value as a sub-slice of the line
returned by bufio.Reader.ReadSlice, which is only valid until the next
read. When the trailer lines arrive in separate TCP segments, reading
x-amz-trailer-signature refills the buffer and overwrites the saved
value, so a correct upload fails with InvalidDigest ("The Content-Md5
you specified is not valid").
The AWS SDK for Java v2 (>= 2.30) on a Linux JDK sends the trailer that
way; about half of its signed streaming uploads failed.
Fixes#11582
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reuse crc32 writer and trim comments in trailer split test
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer sink: keep the gRPC status inside wrapped errors
%v stringifies the status, so a peer teardown reported as Canceled ("the
client connection is closing") reached IsTransientError as plain text and
matched nothing: the sync job failed on the first attempt and pinned the
offset. %w keeps the status reachable, so the retry runs on a fresh
connection once the target is back.
* pb: let a consumer drop the metadata stream to force a resubscribe
A MetadataProcessor job that exhausts its retries pins the processed
watermark so the event replays on the next subscribe — but nothing on the
source stream notices a target-side failure, so the replay waited for an
unrelated reconnect or a restart. The new Resubscribe channel cancels the
stream's context; the Recv loop answers it with ErrResubscribe so the
caller's retry loop resubscribes from GetResumeTsNs and replays the pinned
events in order.
* pb: stop the event retry loop once the stream context is done
RetryUntil ignores context, so a subscriber parked on a failing offset
write would keep retrying past a resubscribe signal until the sink came
back. Stop retrying when the stream is being dropped so the resubscribe
takes effect promptly.
* filer.sync: signal resubscribe when a job failure pins the offset
A job that exhausts its in-job retries leaves the event pinned behind oldestFailedTsNs, replayable only on a reconnect. Closing resubscribeCh on the first recorded failure lets the metadata follower drop the stream so the reconnect replays the pinned events instead of waiting for a process restart (#11572).
* filer.sync: wire the resubscribe signal into the follow options
filer.sync, filer.remote.sync, and the remote gateway bucket sync all run their subscription inside an outer retry loop, so ErrResubscribe resurfaces as a resubscribe from the persisted watermark.
* filer.sync: wait for in-flight jobs before signaling resubscribe
* remote sync: never resume past the saved offset when -timeAgo is set
* filer.sync: drop events that arrive after the drain signals resubscribe
* pb: interrupt the event retry backoff when the stream context ends
* filer.sync: stop admitting once a failure pins, and count jobs per timestamp
A pinned watermark only released once the processor went fully quiet, so a busy stream could starve the resubscribe — the failed event would wait for an unrelated reconnect anyway, the wait this mechanism exists to remove. The processor now latches stopped when a job fails: admission drops new events (they replay from the pinned watermark after the reconnect), a broadcast releases blocked waiters, and the resubscribe signals as soon as the jobs already in flight drain. A redelivery of an event still in the failure ledger may still run so its success shrinks the replay, but nothing starts once the signal has fired, or it would race the replay it asked for.
Dropped events no longer inflate the received counters — an event counts only once admitted, and the replay's own admission counts it.
While here: activeJobs keyed by TsNs collapsed events sharing a timestamp, so one completion could empty the map while a same-ts sibling was still running — letting the drain gate and the watermark outrun it. Jobs are now counted per timestamp, and the drain and lazy heap cleanup go through the counts.
* volume server: refuse the rest of a grouped run after a durable index failure
A durable write whose needle-map put fails stops the volume taking writes
(#10825): sent on its own, the next write then fails read only before it
appends. The grouped run from #11543 appends and syncs every entry before
publishing any, then kept publishing the entries after the failed one and
acked them once the shared .idx sync went through. When the failed put
tore its .idx row, the rows appended after it land off alignment, so the
next load parses them as garbage and the acked writes are gone.
Once a durable entry fails to publish, refuse every later entry of the run
with ReadOnly, as the per-needle path does. The entries before it stay
acked; their rows go down with the run's one .idx sync. The refused
records stay on the .dat unindexed, as the failed one does on its own.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: refuse a grouped entry staged as a cookie mismatch too
After a durable entry in a grouped run fails to index, the entries
after it are refused as they would be on their own. On its own an entry
meets check_writable before its cookie check, so one staged as a cookie
mismatch now gets the refusal too, instead of keeping its staging error.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: trim a torn .idx row back so the next stays aligned
A failed write_index_entry can leave half a row in the .idx. With the
writer appending at the tail, every row written after it lands off
alignment and the next load parses them as garbage, so a write acked
behind a torn row does not come back. Trim the file back to
idx_file_offset on a failed append, in both needle maps, and cover it
with a test that writes past a torn row and reloads.
* volume server: refuse queued Go writes once a durable index update fails
processBatch kept writing after a failed nm.Put, and the single-write
path checked IsReadOnly only outside the volume lock. A durable write
whose index update fails now marks the volume noWriteOrDelete, and each
queued request is checked before it appends, so the ones after a failed
durable entry are refused the way a lone write is. Deletes get the same
noWriteOrDelete refusal a lone delete gets.
* volume server: refuse appends while a torn .idx row cannot be trimmed
When trimming back a half-written .idx row itself fails, the next append
would land after the torn bytes and every later row would parse off
alignment on load. Latch the map as torn and refuse appends until the
trim succeeds, on both CompactNeedleMap and RedbNeedleMap; the same
latch covers an orphan row that could not be trimmed after a failed
redb commit.
The .idx writer is now opened with write+append access so truncate_to
(set_len) works on Windows, where an append-only handle cannot trim.
* volume server: write .idx rows at idx_file_offset, not via append mode
Rust's OpenOptions on Windows strips FILE_WRITE_DATA whenever append is
set so the handle stays strictly append-only, which makes set_len fail -
the torn-row trim could never succeed there. Open the .idx writer with
plain write access and seek to idx_file_offset before each row, the same
positioned-write model the Go server uses.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: skip UpdateEntry when inline content is unchanged
SaveInsideFiler rewrites config files (IAM identities, filer.conf,
remote mappings, policies) unconditionally. Each no-op UpdateEntry is a
metadata event the local meta log persists to /topics/.system/log,
which appends a chunk to a volume. A client that rewrites identical
config on a timer, e.g. the seaweedfs-operator 5-minute resync calling
UpdateUser with unchanged actions, keeps .dat/.idx files growing on an
otherwise idle cluster and prevents HDD spindown
(seaweedfs/seaweedfs#11571).
Skip the UpdateEntry when the stored inline content is byte-identical,
so unchanged writes produce no metadata event and no volume writes.
* filer: test that identical SaveInsideFiler writes skip UpdateEntry
* filer: require stamped Md5 before skipping identical writes
An entry holding identical content but no Md5 (written before hashing,
or by a tool that cleared it) would never get the stamp that
IF_ETAG_MATCH conditional writes key off. Skip only when both the
stored content and its Md5 match, so one write still lands to repair
the stamp.
* filer: resume metadata subscriber from processed watermark on reconnect
* filer: take the reconnect position from GetResumeTsNs verbatim
The callback is the subscriber's durable resume point; falling back to
StartTsNs when it returns zero can resume from a cursor the log-chunk
reader advanced past still-pending work.
* filer: advance the stream cursor once a retried event recovers
RetryForeverOnError resolves the failure inside handleErr, so returning
without moving StartTsNs replays work the event already did when the
stream reconnects before the next one arrives.
* filer: let filtered-progress markers move the processed watermark
A marker means the source examined everything up to its timestamp and
skipped what did not match the subscription. With a resume callback the
marker now reaches the consumer, and AddSyncJob advances the watermark
to it once every earlier job finished and no failure pins the offset.
Idle filtered stretches no longer rescan on every reconnect, while the
guards keep the watermark behind pending or failed work.
* filer: unpin the watermark once a failed event completes
oldestFailedTsNs was only ever set, so a failure that a replay later
fixed still held the resume offset, and every reconnect re-read the
same backlog. Track outstanding failures in a set and recompute the
pin when the failed event's job finally succeeds.
* filer.remote.gateway: resume bucket sync from the processed watermark
The bucket-sync subscriber runs the same MetadataProcessor queue as
filer.remote.sync; give it the same GetResumeTsNs callback so a
reconnect resumes from durably processed work, not the last seen event.
* util: treat a peer-sent gRPC Canceled as transient
A peer tearing down its end of the transport reports codes.Canceled
("the client connection is closing"), which IsTransientError used to
reject: the sync job then failed on the first try and held the offset
until a restart. Caller's own cancels are still excluded up front by
errors.Is(err, context.Canceled), so only teardown-style statuses take
the new branch.
* fix: preserve filtered progress and distinguish caller cancellation
* filer: bound the failed-event ledger past a persistent outage
A destination rejecting every event grew failedTs by one entry per source
event for the life of the processor. Past maxFailedSyncEvents the set now
collapses to a sticky pin at the smallest failure seen, so the watermark
still replays from the oldest failure while memory stays bounded; a
restart re-derives the exact set.
Also keep a resume-callback consumer's chunk-ref replay filter at the
subscribe-time position instead of option.StartTsNs, so a resubscribe does
not filter out events whose async processing is still pending.
* filer: key the failed-event ledger by event, not just timestamp
A success for one event cleared the pin recorded for a different event
that shared its TsNs, letting the watermark pass an unresolved failure.
The ledger now keys on the event's path identity, so recovery unblocks
only the event that actually failed.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume server: read GET/HEAD needles off the store lock, and only once
The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.
The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stream needle chunks without the store lock
StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.
The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.
The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: split get_or_head_handler_inner into phases
get_or_head_handler_inner was a ~650-line function. Its middle resolved
the needle and set five mutable flags (stream_info, can_stream,
can_handle_head_from_meta, can_handle_range_from_source, bypass_cm) that
three if-let reply paths then re-tested, each re-checking stream_info.
It is now a 126-line orchestrator over named phases: reject_read_jwt,
proxy_missing_volume, wait_for_download_slot, parse_read_request,
read_ec_needle / read_volume_needle, etag_and_last_modified,
not_modified_response, read_response_headers, and the reply phases
stream_response, head_from_meta_response, range_from_source_response,
buffered_payload and buffered_response. The read phases return a
ReadPlan whose ReadStrategy enum (Stream, HeadFromMeta, RangeFromSource,
Buffered) carries the NeedleStreamInfo only on the variants that use it,
so the reply is one match instead of three flag checks.
Pure refactor: every status code, header and header order, error text,
metric increment, lock and data-file lease scope, spawn_blocking
boundary and side-effect order is unchanged. Phases that can end the
request return ControlFlow<Response, T>. A Range header that is not
visible ASCII still falls through to the buffered path, as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stop a needle stream once its volume becomes unavailable
Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.
The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: mirror Go order in the buffered read path
- check HEAD before Range in buffered_response (writeResponseContent
order); an EC-volume HEAD with a Range header answered 206, Go
answers 200
- treat the proxied flag as an exact query pair like Go's parsed
lookup, not a substring
- name the phases after their Go counterparts: check_download_limit
and read_ec_shard_needle; reuse has_replication()
- drop comments that restate the code or cite Go line numbers
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
include always returns a string, so gt compared the rendered quantities
(e.g. "1.2884901888e+11" vs "6.442450944e+10") lexically. Growing a volume
from 60Gi to 100Gi/120Gi or 500Gi to 1Ti emitted no kubectl patch: the
StatefulSet was recreated with the new volumeClaimTemplate but the PVC kept
its old size. Shrinks such as 120Gi -> 60Gi emitted a patch instead.
Pipe both values through float64 before comparing.
2026-10-03 15:07:32 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep a ranged read in random mode through its contiguous tail
A far ReadAt on a fresh ReaderPattern left the sequential counter at -1,
so the next buffer of the same ranged request landed on the frontier and
flipped the verdict straight back to sequential — readChunkSliceAt then
paid a whole-chunk fetch for the remainder of the range. Drop the
counter to -ModeChangeLimit when random mode is entered so the verdict
needs sustained sequential evidence to undo, matching the hysteresis an
established sequential stream already gets.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: pin small ranged GETs to range reads
A ranged GET whose first read lands within SeqTolerance of offset 0 is
judged sequential immediately, and even a far-starting range could flip
back mid-request; either way readChunkSliceAt downloads each covered
chunk in full, multiplying disk reads for small ranged reads (measured
~7x). Pin random mode for ranged requests no larger than SeqTolerance so
all of the request's buffer reads stay range fetches. Larger ranges keep
the dynamic pattern, where whole-chunk fetches amortize.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: fetch only the part of a chunk the view covers
Replaces the PinRandomMode size heuristic with a per-chunk coverage rule.
ViewFromVisibleIntervals already clips chunk views to the request window,
so a view that is not IsFullChunk() is one the request only partially
needs; fetch it as a range regardless of the detected read pattern.
This closes the holes a request-size pin left open: ranges larger than
SeqTolerance no longer revert to whole-chunk downloads once their buffers
look sequential, and ranges that fully cover a chunk keep the shared
whole-chunk path instead of fetching 256KiB slices piecemeal. Prefetch
(MaybeCache) skips clipped views so it cannot amplify a range read either.
PinRandomMode is dropped: no caller needs it once coverage drives the
fetch choice. Range fetches route through fetchChunkDataFn so tests
observe them the same way as whole-chunk downloads.
* filer: keep ciphered chunks on the whole-chunk path
A range fetch cannot save bytes for a ciphered chunk: readEncryptedUrl
always downloads and decrypts the whole blob before slicing. Sending
partial views of ciphered chunks through fetchChunkRange would repeat the
full download per buffer, so they keep the shared whole-chunk path where
one download serves every buffer. Prefetch stays enabled for them for
the same reason.
* filer: keep compressed chunks on the whole-chunk path
Like ciphered chunks, a range request on a compressed chunk makes the
volume server read and decompress the whole needle, so range-per-buffer
would repeat the full backend read for each 256KiB window. Route them
through the shared whole-chunk path via ChunkView.CanRangeFetch.
* filer: fall back to range fetch when a chunk exceeds the reader budget
A ciphered or compressed chunk larger than readerCacheSizeMB can never
be read through the whole-chunk path — the budget rejects the buffer —
so its partial views must still range-fetch or the GET fails outright.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: ec.decode reads the .ecx from the index dir it was copied to
VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so
with a split data/index dir the decode target has no .ecx beside its
shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but
built the .idx from the data dir, failing with NotFound after the .dat
was already published. It now reads .ecx/.ecj from where the EC volume
opened them and writes the .idx beside the .dat, where Go leaves it.
The live-entry check and the .dat size also ignored deletions recorded
only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first:
a fully deleted volume was decoded instead of reported as having no live
entries, and deleted tail needles were copied into the .dat. Both now
treat journaled ids as deleted, without rewriting the sealed .ecx.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: ec.decode keeps the decoded volume writable and reads every .ecj
The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after
the .dat was cut short before it, so the mount saw a row past EOF and
marked the decoded volume read-only. Rows of deleted needles the .dat no
longer holds are now dropped, and each journaled needle still in the .dat
gets one tombstone instead of one per journal entry.
VolumeEcShardsCopy appends journals collected from other holders into
the idx dir, but the decode read only the .ecj beside the .ecx, which
sits in the data dir when this server generated the shards. It now
reads both, once, in bounded chunks via the loader EcVolume uses.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: test ec.decode drops a sealed .ecx tail tombstone
Covers the other half of the rule added in the previous commit: a tail
needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from
the .dat, and its row must not reach the rebuilt .idx either.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: ec.decode runs its file I/O off the async runtime
VolumeEcShardsToVolume released the store lock before decoding, but read
the .ecx/.ecj, rebuilt the .dat and wrote the .idx inside the async
handler, parking a runtime worker for the length of a volume-sized copy.
The decode now runs in spawn_blocking on inputs snapshotted under the
store lock.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: ec.decode checks the rebuilt .dat is complete
Go stats the decoded .dat before writing the .idx (VerifyDecodedDatFile)
and fails the decode when it is shorter than the extent the EC index
references, since the caller deletes the shards once the call returns.
The Rust handler returned success without that check. The rebuild
already fails on a short shard read, so this guards the published file
itself.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: ec.decode drops the decoded volume's bitrot sidecars
Go removes <base>.ecsum and <base>.ecsum.v<N> beside the .dat and beside
the .ecx once the .idx is written, so a stale checksum sidecar cannot
pass for the protection of a later re-encode. The Rust handler left them
in place. Removal is best effort, as in Go.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: ec.decode compacts the decoded volume
Go ends VolumeEcShardsToVolume with an offline CompactVolumeFiles, so the
decoded volume holds only live needles. The Rust decode left every needle
deleted through the .ecj in the .dat, tombstoned in the .idx, until a
later vacuum reclaimed it.
Store::compact_volume_files loads the unmounted volume, checks free space
the way the vacuum does (the estimate now lives in one helper), and runs
the vacuum's compact-by-index and commit. As in Go a failed compaction is
logged and the decode still succeeds, so the uncompacted .idx rules stay:
the tests that pin them now make the compaction fail.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: ec.decode keeps deletes journaled while the .dat is written
The decode read the .ecj journals once, before rebuilding the .dat, so a
delete that reached the EC volume during the rebuild was left out of the
new .idx and the needle came back live. Each journal's read length is now
kept, and the bytes appended since are read just before the .idx is
written, after waiting out any journal append in flight (appends hold
the store write lock), so every delete acknowledged by then is in the
.idx. A delete after that point is still lost, as in Go.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Guard overlapping ec decode requests; serialize journal catch-up
volume_ec_shards_to_volume runs its decode in spawn_blocking, so a
dropped request leaves the job running and a retry would race it on the
temporary and final volume files. Claim the vid in a per-server
in-flight set until the blocking job finishes, and return Unavailable
to an overlapping request. The Go handler has the same exposure and
gets the same guard.
Journal appends hold the store write lock through their
sync-or-truncate, so holding a read lock across the catch-up read
guarantees every record it sees is committed: a rolled-back delete can
no longer leave a tombstone in the decoded index.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Reconcile the swap when offline compaction commit fails
A CommitCompact that fails after the .cpc marker may have renamed .dat
but not .idx. cleanup_compact refuses while the marker exists, so the
mismatched pair survived until a restart reconciled it — and the decode
caller treats the failure as non-fatal. Run reconcileCompactState on
commit failure so a decided swap rolls forward and orphan temps are
removed before the volume can mount.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Release the decode claim on panic
* volume: add ec_decodes_in_flight to the integration-test state literal
* volume server: hold the decode tail's lock through compaction
The catch_up read released before the rebuilt .idx was written and the
volume compacted, so a delete synced to .ecj in that window was durably
journaled yet absent from the published index — resurrecting the needle.
Rust now holds the store read lock from catch_up through compact, and Go
mirrors it by holding the volume's journal lock from the journal-
consuming index write through CompactVolumeFiles.
* volume server: serialize ec decode's tail per volume, not per store
Review follow-ups on the decode path:
- Rust: holding the store read lock from journal catch-up through the
offline compaction stalled every writer on unrelated volumes for the
whole rewrite. The new ec_decode_tail set marks the vid only while its
.idx is published and .cpd/.cpx swapped; the two local .ecj append paths
(VolumeEcBlobDelete, the distributed delete's local journal) wait on a
Notify for that span — Go's per-volume ecjFileAccessLock semantics
without the global stall. VolumeMount and the staged-adopt path are also
held off while a decode claim is in flight so neither can race the swap.
- Rust: the initial journal read ran unlocked, so bytes a rolled-back
append later truncated could be folded in as phantom tombstones. The
first pass stays unlocked (a slow journal must not stall the store) and
a rescan under the quiescing read lock re-reads only committed content;
catch_up now rebuilds the id set when a regular journal shrank.
- Go: the decode resolved the compaction DiskLocation through
FindEcVolume while holding the journal lock, inverting DestroyEcVolume's
map->journal order into a deadlock. The lookup now happens first, and
DestroyEcVolume/deleteEcVolumeById/DiskLocation.Close destroy outside
the map lock.
- Go: RebuildEcxFile unlinks .ecj while the volume's ecjFile handle stays
open, so later deletes could commit to a detached inode. Both call sites
now fold under the journal lock and ReopenDeletionJournal repoints the
handle at the live path, working on the volume's resolved .ecx dir
(EcIndexBaseFileName) rather than the configured index dir.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: fence EC remounts behind the destroy tombstone
DestroyEcVolume, deleteEcVolumeById, and the collection-delete sweep now
remove the EcVolume from ecVolumes before destroying it off-lock, so a
concurrent remount could re-open shard files that the in-flight destroy
then unlinks — registering a detached fd.
Each destroy records a per-vid tombstone channel in a new
ecVolumesDestroying map before dropping the map entry and closes it when
Destroy returns. The tombstone intentionally survives as the vid's
destroy generation: loadEcShardWithIdxDir compares it before and after
opening the shard, so a destroy that both started and finished inside the
open window is still detected. A mismatch drops the just-opened shard
(releasing its fd and mount gauge) and retries after the destroy
completes; a successful mount clears the stale tombstone.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: rescan the .ecj under the store lock only after a rollback
The decode's second journal pass ran a full rescan under the store read
lock on every decode, stalling unrelated writers for the length of the
scan. Bump a process-wide epoch whenever a failed append truncates its
uncommitted tail; an unchanged epoch between the unlocked read and the
quiesced pass proves every id folded in was committed, so catch_up()
suffices. catch_up() also treats a journal that was read but has since
disappeared as shrunk to zero, so its earlier ids cannot linger.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: check the decode tail under the store write lock on delete
A blob delete waited for the publishing tail before taking the store
write lock, so a decode that claimed the tail while the delete was
parked behind the decoder's read lock could still see the journal append
land after the rebuilt .idx — an acknowledged delete the mount would
miss. Test tail membership under the write lock instead, retrying after
the wait; journal_delete_local reports WouldBlock for the same recheck
on the distributed path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: claim the vid for mount and staged adoption, per volume
VolumeMount and the staged .copying adoption held the
ec_decodes_in_flight set lock through slow file renames and mounts,
stalling every unrelated volume's decode, mount, and adoption. Take the
per-volume claim instead — the same exclusion against a racing decode
for this vid, released when the call returns.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: fail the decode when a compaction commit marker survives
CompactVolumeFiles' caller logged a compaction error and went on to
delete the EC shards. When the commit marker (.cpc) is still on disk the
.dat/.idx swap was decided but could not be reconciled, so the mounted
pair may be mismatched — report the failure instead so the shards are
kept and the caller can retry.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: gate the parked-delete test on the held write lock
The releaser thread and the spawned delete raced for the store write
lock; on a slow runner the delete could acquire it first and commit
before the tail was ever claimed, failing !delete.is_finished() on the
Windows unit-test job. Spawn the delete only after the thread reports
the lock held.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: do not pin the sync offset on completed work
* filer.remote.sync: a superseded rename uploads the current entry; typed NotFound for a stamp on a deleted entry
* filer.remote.sync: a superseded rename keeps the old key when it is the only copy and uploads once
* filer.remote.sync: a rename whose content is now remote-only fails the event instead of completing it
* filer.remote.sync: a remote-only rename copies the old object to the destination before deleting it
* filer.remote.sync: the remote-only rename path follows the filer's current entry and verifies the destination object
* filer.remote.sync: an event that described an entry without data is superseded once the filer wrote to it
* filer.remote.sync: a superseded rename does only the work left to do
uploadCurrentEntry met a remote-only current entry with a fixed error, but a
sync plus remote.uncache in the meantime leaves the destination holding the
stamped object; that state is complete, not lost. The remote-only case now
finishes through completeRemoteOnlyRename, which verifies the destination
against the entry stamp and fails only when neither key holds the content.
A current entry whose stamp covers its content was already uploaded by the
superseding event; skip it instead of writing the same bytes again.
* filer.remote.sync: an inherited stamp does not prove the content synced
The stamp-coverage skip in uploadCurrentEntry read LastLocalSyncTsNs as
proof the current content was uploaded, but a rename carries the source
entry's stamp to the destination: a rewrite hidden by that stamp (the case
the fallback upload exists for) carries a LastLocalSyncTsNs at or after its
mtime and would have been skipped. Drop the check; the remote-only path
verifies content at the destination itself through describes.
---------
Co-authored-by: James Sas <james@medable.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3api: allow unsigned SSE-C customer key headers on presigned requests
AWS requires only x-amz-server-side-encryption-customer-algorithm to be signed on presigned URLs; the key and key-MD5 headers are supplied at request time. Since #9121 rejected any x-amz-* header outside SignedHeaders, SDK-generated presigned SSE-C requests (e.g. .NET GetPreSignedUrlRequest) fail with SignatureDoesNotMatch. Exempt the customer key and copy-source key headers for presigned requests only.
* s3api: test presigned SSE-C requests carrying unsigned key headers
* vacuum: bound the commit RPC with a phase deadline
VacuumVolumeCommit ran on context.Background(), so a volume server that
keeps the call pending would hold the topology-wide vacuum guard
forever and every later sweep would be skipped. Give the call a
deadline scaled like the existing phase waits (one minute per GB of
the volume size limit) so a stalled commit ends as an error instead of
blocking the sweep; the timeout is a var so tests can shrink it.
* vacuum: bound the replica status probe with a phase deadline
The VolumeStatus call on replicas that were not compacted also ran on
context.Background(), so a stalled replica could pin the sweep the
same way a stalled commit can. Give it the same per-phase deadline.
* vacuum: bound the cleanup RPC with a phase deadline
VacuumVolumeCleanup also ran on context.Background(); a stalled
server would keep the sweep worker and the shared vacuum guard
pending forever. Give it the same per-phase deadline.
* vacuum: let the check and compact phase waits cancel their RPCs
The coordinator wait timers fired while the check and compact calls
still ran on context.Background(), so the sweep gave up but the RPC
goroutine stayed until the server answered, and a compact stream kept
writing on the server. Share one deadline context between the wait and
the calls so an expired wait actually cancels them.
* vacuum: test that a stalled volume server releases the vacuum guard
A fake volume server keeps one vacuum-phase RPC pending until the
client context is cancelled. Before the phase deadlines, Vacuum never
returned and vacuumLockCounter stayed held; now each phase cancels on
its deadline and the guard is free for the next request.
* volume: stop compaction at the next needle when the client cancels
The progress callback only noticed a gone client when a 128 MiB report
failed to send, so an aborted VacuumVolumeCompact kept copying for up
to a whole interval while the master had already moved on to cleanup.
Check the stream context on every needle, the same early return the
Rust volume server does with tx.is_closed().
* vacuum: assert the stalled phase RPC is cancelled, not just bypassed
The check and compact coordinator waits already returned on timeout
before the deadlines existed, so a regression that put the calls back
on context.Background() would pass unnoticed. Wait for the fake server
to report that the phase RPC context ended.
* vacuum: give the stalled-RPC test room to reach the handler
The 50ms phase budget starts before goroutine scheduling and the gRPC
dial, so a busy test host could expire it before the fake server saw
the call. Raise the override to 250ms; the test still finishes in
about a second.
* vacuum: describe the phase deadline as scaled, not per-GB
The formula keeps the exact expression the check and compact waits
already used (floor plus one at 1 GiB granularity); it is a backstop,
not a per-GB SLO.
AsInode derives inodes from HashStringToLong, which is uniform over int64,
so roughly half of the derived values land above math.MaxInt64 once they are
converted to uint64. The Elasticsearch store indexes Entry.Attr.Inode as a
signed long, so those values are rejected with HTTP 400 and the metadata
entry is never written, which the filer then retries forever.
Fold the sign bit off in one place, util.NormalizeInode, and route both
derivation sites through it: FullPath.AsInode (path plus creation time) and
the hard-link branch in ensureEntryInode (HardLinkId hash). Masking keeps
the other 63 hash bits, so distinct paths still get distinct inodes, and it
applies identically to the FUSE mount, which derives the same value.
Co-authored-by: Yi-111-a <34116709+0-xiaosu@users.noreply.github.com>
* fix(s3): track filer joins and leaves pushed by the master
The S3 FilerClient replaced its -filer seed with a master snapshot of
filer IPs at boot and refreshed it only every 5 minutes. A rolling
restart replaces every filer well inside that window, leaving S3
servers with only dead addresses and failing every write until the
next poll.
Apply the master's ClusterNodeUpdate pushes to the filer list as they
arrive, keeping the poll as a backstop. The last filer is never
removed, and a poll snapshot requested before a push was applied is
discarded rather than overwriting newer membership.
* Defer last-filer leaves; bump the generation only on real changes
* fix(s3): cancel deferred filer leaves on rejoin and on discovery
A deferred last-filer leave outlived the filer it was recorded for: a
rejoin at the same address looked like a duplicate add, and a discovery
snapshot left the entry behind. The next join then removed a live
filer until the following poll.
A join now cancels any deferred leave for its address, and an applied
snapshot clears them, since it is the master's current membership.
* Bump the push generation when a rejoin cancels a deferred leave
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: compact an oversized .ecj at mount, safely (Rust + Go)
Restore the mount-time compaction dropped from #11408, Rust + Go parity.
A journal already bloated by repeated shard copies is folded down to the
id set it encodes.
- Trigger after load when file_records > max(threshold, 4x distinct),
with a 1 MiB floor so small journals are never rewritten. The set is
written to .ecj.compact.tmp + fsync, the handle dropped, renamed,
the directory fsynced and the append handle reopened. A failure before
the rename keeps the original journal and handle; a failure after it
fails the mount.
- Go never compacts after a failed journal load; the set would be
partial and the rewrite would drop the unread records.
- A per-path registry (ecj_registry.rs / ecj_registry.go) counts EcVolume
holders and out-of-band writers of each .ecj. Compaction runs only
when this volume is the sole holder and no copy is writing; holders
and writers wait while one runs. This covers shared -dir.idx journals
and cross-disk reconcile, where another EcVolume may hold the same
journal.
- VolumeEcShardsCopy and EC index recovery register as writers around
their .ecj append and partial-file cleanup.
- Under the reservation, re-check that the file on disk is still the
inode and size that was loaded.
- Publish errors are classified where they happen; a failed rename plus
a failed restore reports both errors.
- Compaction runs after the .vif / bitrot checks, so a refused mount
leaves the journal untouched.
- The tmp is opened like other volume files, removed at mount if a crash
left it, and listed in every EC index cleanup path.
Failure paths are tested through the real mount via injectable fs steps
(open_with / newEcVolumeWith), plus sibling holders, active copies,
changed-after-load, stale tmp cleanup, refused mounts and the Go
load-error guard.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: fail the mount when the compacted .ecj's directory cannot be synced
The Rust mount synced the journal's directory after renaming the compacted
file over it through the crate's best-effort fsync_dir, which returns Ok
when the directory cannot be opened. A rename needs only write and search
permission, so on a directory without read permission the replacement was
published, never synced, and the mount went on taking deletes against it.
Sync through a helper that propagates the open error, as Go's
util.FsyncDir already does, so that case fails the mount like any other
post-rename sync failure.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: test the no-compaction-after-failed-load rule through the Go mount
The test for it handed compactEcjAfterLoad an artificial error on a volume
that had loaded cleanly, so it would not notice NewEcVolume dropping the
real load error on the way to compaction.
Make the journal read one of the injectable ecjFsOps steps and fail it
inside the real mount, after the first chunk, on a journal whose last
entry is an id the first chunk does not hold. The mount must leave the
file byte for byte as it was; a clean remount then compacts and keeps
that id. The Rust mount fails outright on a load error, so it has no
equivalent path.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: register ReceiveFile's .ecj writes with the journal registry
ReceiveFile refuses a mounted EC volume only once, when the info message
arrives, then creates the .ecj and streams chunks into it. A volume that
mounted on that journal mid-stream could find a bloated prefix, pass the
inode-and-size re-check and rename a compacted file over it; the rest of
the stream then went to the unlinked inode and was lost.
Register the path as a writer before the file is created, in both the Go
and Rust handlers, and hold it until the file is closed and any partial
copy removed, as the shard-copy and index-recovery appends already do.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: skip .ecj compaction when a writer ran since the journal was loaded
Compaction checked only that no writer was active at the reservation, and
that the file was still the loaded inode at the loaded size. A ReceiveFile
truncates and refills the journal in place, so one that ran during the
mount's load, or after it, and finished before the reservation could leave
different ids at the same length; compaction then wrote the stale set over
them.
Give each path a write generation that every writer bumps as it starts. A
holder records it, and whether a writer was active, when it registers,
which is before it opens and loads the journal. It may compact only if no
writer was active then and the generation has not moved. Same rule in Go
and Rust; the journal read becomes an injectable step in Rust as it is in
Go, so both test the in-place rewrite through the real mount.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: match the ReadOnly(VolumeId) variant in write_volume_needles
#11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3/iam: manage roles through the IAM API, with an opt-in persistent role store
Roles could only come from the IAM config file: the S3 server pinned the
role store to memory and the embedded IAM API had no role actions, so a
role could not be created, retrusted or revoked without editing the file
and restarting every gateway.
Role store
- Read the `roleStore` key (the IAMConfig field already existed). With an
IAM config file the default stays memory; with none it is the filer, as
for OIDC providers, so zero-config clusters keep runtime-created roles.
- Roles from the IAM config file never go into a persistent role store,
which outlives the file and may be shared by S3 servers with different
files. They are served from memory beneath the store, as OIDC providers
are: a stored role of the same name takes precedence, and deleting it
restores the file's. A config-file role cannot be changed or deleted
through the API (UnmodifiableEntity), and removing one from the file
removes it at the next start. An in-memory store holds them as records,
as before. They have no creation time, so CreateDate is omitted rather
than reporting when this server started. SetRoleStore installs a store
the same way, so a store set after startup keeps the config-file roles,
as SetOIDCProviderStore does for providers.
- Watch /etc/iam/roles and drop the cached role definitions on change. The
cached filer store otherwise serves a peer's stale role for up to its 5m
TTL, which keeps a revoked trust policy in force on the other gateways.
- Role stores wrap ErrRoleNotFound for a missing role; the filer store
used to report any failed lookup as "role not found". CreateRole proceeds
only on a confirmed absence, so an unreadable store cannot let it write
over an existing role.
IAM actions
- CreateRole, GetRole, ListRoles, DeleteRole, UpdateAssumeRolePolicy,
AttachRolePolicy, DetachRolePolicy, ListAttachedRolePolicies. The reads
are allowed in read-only mode.
- A role defined in the config file is reloaded from it at every start, so
changing or deleting it through the API is refused (UnmodifiableEntity)
rather than silently reverted.
- DeleteRole with policies attached is refused (DeleteConflict), as on AWS.
- Role names follow AWS's rules ([\w+=,.@-]{1,64}); a role is stored as
<name>.json in the filer, so this also keeps a name from leaving the role
store's directory. At most 10 managed policies per role (AWS's default
quota; MaxManagedPoliciesPerUser is 10 too), LimitExceeded beyond.
- DeletePolicy is refused (DeleteConflict) while a role attaches the
policy, as it already is for users and groups: roles attach policies by
name, so a policy created later under the deleted one's name would
otherwise take effect on the role.
- Role paths other than "/" and role tags are not stored, so they are
refused rather than dropped.
Role IDs and sessions
- Roles get a unique RoleId when first stored (random, AWS AROA form),
kept across updates; a config-file role gets a stable ID derived from its
name, since it is created again at every start.
- Sessions issued through AssumeRoleWithWebIdentity, AssumeRoleWithCredentials
and AssumeRole carry the role's ID (claim "rid"), and a request under a role
whose current ID differs is denied. Resolving a session's policies by role
name let a session outlive its role: once a role was deleted, a role later
created under the same name — with a different trust policy and different
policies — revived every unexpired session of the old one with the new
role's permissions. Sessions issued before this change carry no ID and are
unaffected until they expire.
Integration test (test/s3/iam, run with `make start-services`):
TestWebIdentityWithProviderAndRoleManagedThroughIAMAPI configures an OIDC
provider, a managed policy and a role entirely through the IAM API against a
JWKS served by the test, then checks the trusted subject gets credentials
scoped to the attached policy; another subject, a token signed by another
key, an unsigned token and a token for another audience are refused; and UpdateAssumeRolePolicy moves the
trust at once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: manage OIDC providers and roles over the filer IAM gRPC service
The filer's SeaweedIdentityAccessManagement service covers users, access
keys, policies and service accounts, but not the OIDC providers and roles
that STS web-identity federation needs. A controller that already manages
IAM over this service (seaweedfs-operator's S3OIDCProvider) has no
transport for them; its swadmin client returns ErrOIDCNotWired and names
this as the recommended fix.
- PutOIDCProvider / GetOIDCProvider / DeleteOIDCProvider / ListOIDCProviders
and PutRole / GetRole / DeleteRole / ListRoles.
- They write the filer-backed stores at their default paths, which S3
servers read when configured with a filer-typed "oidcProviderStore" and
"roleStore"; the S3 servers' /etc/iam subscription applies changes
without a restart.
- Put is an upsert, so a controller can reconcile to it. Deleting a
provider or role that does not exist returns NotFound, as DeleteUser does
for a user; clients treat that as already deleted. The provider's account
ID travels in the request, since the filer does not know the STS
accountId.
- PutRole applies the IAM API's rules: AWS role names, at most 10 managed
policies.
- An S3 server serves the roles and providers of its own IAM config file
ahead of the store, so a stored entry with the same name has no effect
on that server.
- PutRole keeps a replaced role's RoleId and gives a role created anew a
fresh one, so sessions of a deleted role do not carry over to a later role
of the same name.
- DeletePolicy returns FailedPrecondition while a role attaches the policy
(see the IAM API's DeleteConflict in the previous change). DeletePolicy on
this service still does not check user attachments, which predates this.
- PutOIDCProvider requires an https issuer (http only for a loopback host):
STS fetches the issuer's signing keys from it, so over plain HTTP anyone
on the network path could substitute their own.
- The OIDC provider and role RPCs refuse to run on an unauthenticated
service (FailedPrecondition until jwt.filer_signing.key is set). Users and
policies keep the service's opt-in auth, but these grant STS access
outright: otherwise anyone who can reach the port could register an issuer
they control, create a role trusting it, and exchange a token for S3
credentials. The filer's unauthenticated notice becomes a warning that says
so.
- A store that cannot be read is Unavailable, never "not found", so a Put
never writes over an entry it could not see.
- Validation is shared with the IAM API through PrepareRoleDefinition and
PrepareOIDCProviderRecord.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: bind every role session to its role, and change roles atomically
Review follow-ups.
Session binding
- The role-ID check ran only when a session carried no policy names, and
AssumeRole embeds the role's attached policies, so those sessions kept
their permissions after the role was deleted or recreated. The check
now runs for every session carrying a role ID, before policy selection.
- A named role that cannot be resolved at issuance gets no session,
instead of one with no role ID (which nothing binds).
- A config-file role's ID is derived from its name and trust policy, not
the name alone: a different role put in the file under the same name
gets a new ID, while an unchanged role keeps its sessions across restarts.
Role writes
- RoleStore gains UpdateRole, a read-modify-write that lands only if the
role is unchanged since the read, and otherwise re-reads and retries. The
filer store uses the filer's write conditions (IF_NOT_EXISTS for a new
role, IF_ENTRY_EQUAL otherwise). CreateRole, UpdateAssumeRolePolicy and
Attach/DetachRolePolicy all go through it, so two gateways no longer
overwrite each other's changes, a change racing a delete no longer
writes the role back, and of two concurrent creates one gets
EntityAlreadyExists.
- The filer store's ListRoles pages past 1,000 entries and fails on a
broken stream instead of returning what arrived, so DeletePolicy's
attachment check sees every role. ListRoles skips a role deleted between
listing and reading it.
- CreateRole validates first; a failed write is ServiceFailure, not
InvalidInput. Any Tags.* parameter is refused, not only the first key.
- ExecuteAction's skipPersist covers the S3ApiConfiguration only; the
comment now says so. Role and OIDC provider actions write their own stores.
Each fix has a test that fails without it. Against a real filer with two
gateways, concurrent AttachRolePolicy calls lost 1-4 of 8 attachments per
run before this change and none after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: PutRole changes roles atomically and checks its ARN; https issuers' keys stay on https
Review follow-ups on top of the role-store changes.
- PutRole goes through RoleStore.UpdateRole, so the decision to keep an
existing role's ID or mint a new one is made against the role as it is
when written. A PutRole racing a DeleteRole can no longer write the
deleted role back with its old ID, which would revive its sessions. A
failed store read or write is Unavailable.
- PutRole refuses a role_arn that does not name the role: STS resolves a
role by the name in the ARN it is given.
- PutOIDCProvider requires an https issuer, but discovery could still name
a plain-http jwks_uri, and a key fetch could be redirected to http. For
an https issuer, a non-https jwks_uri from discovery is refused (the
issuer's own /.well-known/jwks.json is used instead), and the client
that fetches discovery and keys refuses any https-to-http redirect. An
operator-set jwksUri is left as configured.
Each has a test that fails without its guard.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: one role snapshot per decision; DeleteRole is atomic; watch a custom role store path
Review follow-ups.
- Authorization evaluates the policies of the role definition the session's
binding was checked against, instead of reading the role again: a role
replaced in between cannot lend a session its policies.
- AssumeRole and AssumeRoleWithLDAPIdentity issue the session from the
definition whose trust admits the caller (IAMManager.ResolveRoleForPrincipal),
and take its ID, duration cap and embedded policies from that same
definition. A role replaced after the caller's trust check by one that does
not trust the caller now yields AccessDenied, not a session bound to the
replacement.
- A RoleUpdate that returns nil deletes the role, on the same condition as a
write: the filer store deletes with ObjectTransaction on IF_ENTRY_EQUAL,
routed and locked like the conditional CreateEntry. DeleteRole decides
against the role it deletes, so a policy attached meanwhile on another
server is a DeleteConflict, and a delete never removes a role written
after its check.
- S3 servers watch the role store's configured basePath, not only
/etc/iam/roles, so a custom path also drops peers' cached roles on change.
Each has a test that fails without it. Live against a real filer: DeleteRole
refuses while a policy is attached and removes the entry once detached; all
test/s3/iam CI stages pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: state which roles DeletePolicy's attachment check can see
RolesAttachingPolicy sees the stored roles and this server's config-file
roles. A role defined only in another server's IAM config file is invisible
to it, so a config-file role that attaches a managed policy is protected
only on the servers whose file defines it. The doc comment now says so and
how to avoid it: keep such roles in every server's file, or attach only
config-file policies to config-file roles.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: note that a role store set after startup is not watched for peer changes
S3 servers build their metadata watch list once, at startup, from the role
store installed then. SetRoleStore's doc now says that a filer-backed store
installed later with a different basePath is not watched, so peers' changes
to it reach this server's cached roles only when the cache expires.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: DeleteRole deletes only the role it saw; issuer URLs are bare
Review follow-ups.
- The filer IAM service's DeleteRole looked the role up, then deleted by
name, so a PutRole landing in between had its new definition deleted. It
now deletes through RoleStore.UpdateRole, conditional on the entry it
read. If the role was replaced meanwhile, it returns Aborted rather than
deleting the replacement, and the caller decides again.
- PutOIDCProvider refuses an issuer URL with userinfo, a query or a
fragment. The provider's ARN comes from host and path alone, while STS
matches a token's iss claim against the stored URL exactly, so such a
provider shared the bare issuer's ARN and matched no token. A loopback
"localhost" is now matched without regard to case.
Both have tests that fail without them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: write OIDC providers atomically over the filer IAM gRPC service
PutOIDCProvider read the record, then stored unconditionally; a racing
DeleteOIDCProvider left the put's stale read merged into the rewritten
record. DeleteOIDCProvider read, then deleted unconditionally; a racing
PutOIDCProvider's newer record could be removed instead. These are the
races the role RPCs closed with UpdateRole.
OIDCProviderStore gains UpdateProvider with the same contract: memory
under its lock, filer as a conditional write (IF_ENTRY_EQUAL /
IF_NOT_EXISTS) or conditional delete retrying a changed entry.
PutOIDCProvider merges the fields the request cannot carry against the
record as it is written; DeleteOIDCProvider aborts rather than delete a
record replaced meanwhile.
isRoleWriteConflict is renamed isEntryWriteConflict — the conditional-
write check is shared by both stores now.
* iam: guard PutRole against a nil credential manager, fix its doc comment
PutRole read attached policies through s.credentialManager without the
nil check its sibling handlers make, so a server built without one
panicked on a PutRole naming a policy. It now fails the call as
FailedPrecondition like the others.
The doc comment also had the store/static precedence backwards: a stored
role shadows a same-named config-file role (as the overlay serves it),
not the other way around.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* vacuum: size the compaction space check by live bytes, not volume size
ensureCompactVolumeSpace required the volume's current .dat and .idx size as
free space before compacting. That is the size of the garbage, not of what
compaction writes, so on a disk that filled up until its volumes went
read-only every compaction was refused, including all-garbage volumes that
would compact to a superblock and an empty index. The sweep then retried
every volume each cycle and reclaimed nothing (issue #11516).
Estimate the output from what the needle map already tracks: live content
bytes plus a per-needle framing upper bound behind a superblock, and one
index entry per live needle. The estimate never exceeds the current volume
size and preallocate still wins when larger. Volumes whose deleted sizes are
unknown (.sdx converted back to .idx) keep the whole volume as the estimate.
The disk probe moves behind a package variable so the tests can stand in
for a full disk; the tests build real volumes instead of re-implementing
the formula.
* vacuum: space check reserves the index on top of preallocate, checks a separate index disk
Review follow-ups: preallocate only stands in for the new .dat, so the
rebuilt index is added on top of it; with separate index directories the
data disk is checked for the .cpd and the index disk for the .cpx; and the
estimates carry 1/16 headroom because counters rebuilt from an index file
pass through a Bloom filter with a 0.1% false positive rate. Neither
estimate exceeds the current file.
* vacuum: split the space check by filesystem, not by directory name
Two directories can sit on one filesystem and share its free space, so
the data and index estimates are checked separately only when the index
directory is on another device; otherwise the sum must fit. Unknown is
treated as shared.
* vacuum: ask the index directory for its share even when it looks like the same filesystem
A volume mounted under the data directory's drive letter on Windows has
the same volume name, so the identity check calls it shared. Checking the
index directory for the index estimate as well costs one statfs and
catches a full index mount either way.
* vacuum: identify a Windows volume by its GUID, not its path prefix
A volume can be reached through a drive letter and through a folder it is
mounted on, so filepath.VolumeName says nothing about the free-space pool.
Resolve each directory to its mount point and compare the volume GUIDs;
when that fails the two are treated as shared.
* vacuum: keep the framing and disk_space_low coverage the rebase displaced
* rust volume: split the compaction space check across data and index disks
Mirror the Go check: estimate the new .dat and rebuilt .idx separately —
live content plus per-needle framing capped at the current file, with
preallocate standing in for the data file when larger — and check each
directory against its own filesystem's free space. Two directories on one
filesystem are asked for the sum.
* vacuum: tighten comments on the compaction space check
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-03 08:31:56 +08:00
Eliah RusinGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Claude Opus 5.5Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: merge .ecj as a set union on EC shard copy + index recovery (Rust+Go)
An EC volume's deletion journal is a set of needle ids, but shard copy
and index recovery appended the peer's whole journal, doubling the file
on every ec_balance round trip. Fold the peer's ids in as a union
instead: only ids the local journal lacks are appended.
- The journal is never replaced. A mounted EcVolume merges a peer's ids
through its live handle under the lock deletes take (Go
MergeJournal / Rust merge_journal), wherever its journal lives.
- An unmounted journal gets only the missing ids appended while mounts
are excluded; the delta is read outside the lock and re-read if the
journal changed.
- The source .ecj streams into memory as an id set: no staging files,
chunked reads, memory proportional to distinct ids.
- Go and Rust agree that a source journal exists when it sends a
modified time or any bytes. A missing source stays a no-op.
- Rust runs every merge in spawn_blocking and shares one receive/merge
path between shard copy and index recovery.
The decode path and the journal format are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: route .ecj merges to the runtime that holds the journal open
Disks sharing one index directory all resolved as the journal's owner, so
the last one won and a sibling's mounted runtime was skipped: the merge
appended behind its open handle and the sibling kept serving the peer's
deleted needles until remount. Callers now name the receiving disk by its
data directory; the merge goes through that disk's runtime, else a
sibling runtime whose journal is the target file.
In Go the unmounted append now holds every disk's EC lock (in location
order) while it rechecks for a mount, so a sibling mounting from this
disk's index during the unlocked read is merged through instead.
In Rust a mount that lands during the read is merged through directly and
its added count returned, rather than discarded and reported as zero.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: sync merged .ecj records outside the disks' EC locks
The unmounted merge held every disk's EC read lock across its fsync, so a
slow sync on one disk held off mounts on all of them, along with the EC
reads queued behind those mounts. Mounts only need to be excluded while
the records are written: the write now happens under the locks and the
fsync after they are released, since a later mount reads the written
records from the page cache. A failed fsync rolls back only if nothing
has mounted the journal or appended to it since the write.
A merge through a mounted volume now keeps only that volume's disk locked
across its fsync.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: roll back an unsynced .ecj merge through a volume mounted mid-sync
If a volume mounted after the unmounted merge wrote its records but before
the fsync failed, the rollback kept the records because the journal was now
open, leaving ids in the volume's deleted set that may never reach disk; a
retried merge then saw them and synced nothing. The rollback now goes
through that volume the way its own failed journal fsync does: truncate
back and drop the ids from the in-memory set, so a retry appends and syncs
them again. It still keeps the records if the volume journaled since, as
truncating would lose that delete. No fsync runs under the disk locks.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: decide .ecj merge rollback from the journal's actual length
Two runtimes can hold one journal (cross-disk mounts). The rollback of an
unsynced merge checked one runtime's cached ecjFileSize, which another
runtime's appends leave stale, so it could truncate a delete that runtime
had already synced. The rollback now holds every holder's journal lock and
truncates only if the file's actual length is still the append's end,
then updates each holder's size and deleted set. Otherwise later records
follow the merged ones, so they stay and are rewritten in place and
synced outside the locks, rather than left possibly not durable.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: keep unsynced .ecj merge ids out of mounted deleted sets
When a merge's fsync failed, later records blocked the rollback, and the
rewrite-and-sync failed as well, the merged ids stayed in every mounted
volume's deleted set without being shown durable, so a retried merge saw
them as present and synced nothing. They now leave those sets while the
records stay in the file, matching DeleteNeedleFromEcx, which publishes an
id only after its record syncs. The merge returns the error and a retry
appends and syncs them again.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: publish merged .ecj ids to every holder of the journal
Two runtimes can journal into the same file when disks share an index
directory. The merge went through only the first holder, leaving a
sibling's in-memory deleted set without the ids, so it could keep
serving a needle the peer deleted until it remounted. Every holder of
the journal now gets the merged ids, in Go and in the volume server.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Publish merged .ecj ids to the journal actually written
mountedEcJournal prefers the receiving disk's own runtime for the vid,
whose journal may live in its data directory while the copied records
name a sibling's journal in the index directory. Publishing by the
requested ecjPath then marked a holder of a different file deleted on
records that file never persisted, resurrecting the needles on remount.
Publish by the picked runtime's journal path instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(filer): persist pending chunk deletions across restarts
The in-memory FileIdDeletionQueue and DeletionRetryQueue lose every
queued-but-unconfirmed deletion when the filer process restarts. Because
deletions only enter the pipeline through that queue, a crash between
enqueue and the volume confirming the delete leaks the chunk permanently:
nothing remembers it. In a multi-filer deployment this was observed as
growing collections of orphaned chunks after filer restarts, and — via
meta-replay from a peer that still had the entry — orphans being
"resurrected" as live references on the recovered filer.
This implements the "periodic snapshot with recovery on startup" option
noted in the existing DeletionRetryQueue TODO, using the store's KV layer
(no new iterator API required across the 15+ store backends):
- queueDeletions() is the single entry point that keeps the hot in-memory
queue and the durable ledger in sync.
- Only terminal outcomes (success / not-found / permanent) remove an id
from the ledger; retryable failures keep it, which is the point.
- A timer and Shutdown() snapshot the pending set to a single KV key.
- On startup, reloadDeletionLedger() re-queues recovered ids after a
grace window so the initial peer meta-aggregation settles first. This
avoids a new hazard: purging a chunk that a lagging peer is about to
re-reference as live data (stale replay turns a stale read into a
dangling read otherwise).
- Volume deletes are idempotent (not-found == success), so re-deleting
after a crash never double-frees.
- Kill switch via viper: filer.deleteQueue.persist=false opts out entirely
(reload also refuses to recover so a stale ledger never comes back).
Tunables: filer.deleteQueue.persistInterval, .recoveryGrace.
Adds unit tests covering snapshot+recover, retry-keeps-entry, disabled
switch, and zero-value Filer safety (run green under -race).
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.8-Flash-Next-ROCmFP4)
* filer: harden the deletion ledger
- Scope the ledger key by filer address so filers sharing one store do
not overwrite each other's pending sets; ledgers written under the
old unscoped key are claimed once on startup.
- Serialize snapshots on deletionSnapshotLock so an in-flight timer
snapshot cannot overwrite a newer shutdown snapshot, and wake the
snapshotter on every queue/forget so a queued id persists within
milliseconds instead of a full interval.
- Merge recovered ids into the pending set immediately on reload; only
the queue push waits out the grace window, so an early snapshot
rewrites the recovered ids rather than dropping them.
- A failed or unparseable ledger read blocks persistence for the run
instead of letting snapshots overwrite the unread ledger.
- Split the ledger into part keys when it exceeds one 64KB value so
stores with a size cap (FoundationDB) do not strand the backlog.
- GetReadyItems reports retry-exhausted ids so they are forgotten in
the ledger instead of replaying after every restart.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: close the remaining deletion-ledger durability gaps
- A manifest referencing a missing part is corruption: surface a wrapped
error and block persistence instead of treating the ledger as absent.
- Multipart snapshots write generation-scoped part keys and publish the
manifest last, so a crash never mixes old and new part contents.
- Orphaned parts are tracked in a persisted .stale sidecar and retried.
- Legacy/index ledgers are republished under the scoped key before the
old keys are removed.
- A ledger index lets a filer restart under a new address claim the
ledger its previous incarnation left behind.
- Expired and permanently-failed retry items only forget the ledger
epoch they recorded, so they cannot erase a re-queued id.
- A failed startup read no longer disables persistence: every snapshot
retries the reload until the store reads again.
* filer: tighten ledger claiming, index updates, and retry epochs
- touchLedgerIndex verifies its write and retries so a concurrent
filer's merge cannot silently drop this key from the index.
- Foreign-ledger claims abort on any unreadable source instead of
leaving it stranded once the new scoped key exists.
- A source that republished during the claim is left in place and its
newer ids merge into the claimant's pending set.
- AddOrUpdate no longer overwrites the ledger epoch of an in-flight
retry item, so its expiry or permanent outcome cannot forget a record
that was re-queued after the attempt began.
- The recovery grace wait exits on shutdown instead of re-queueing
after the filer has stopped.
* filer: requeue surviving records, persist claim deltas, guard index writes
- A dropped retry item (expired or permanent) whose ledger record was
re-enqueued now pushes the id back through the hot queue instead of
leaving it pending with nothing scheduled.
- Ids merged from a claim source that republished mid-claim are
rewritten under our ledger immediately, so they are durable even if
the claimant crashes before the next snapshot.
- touchLedgerIndex aborts when the index read fails for a real error;
only ErrKvNotFound means the index is empty, so a transient failure
can no longer wipe peer entries with a one-key write.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(filer): option to store system metadata logs in their own collection
The filer's internal /topics/.system/log chunks are assigned to the
filer's default collection (-collection). In a multi-filer deployment
that default is often empty, so every restart flap, full-sync, or
event-buffered flush grows the default collection with system chunks that
are indistinguishable from user data in collection.list. This is a large
part of what makes the default collection balloon and confuses orphan
analysis.
This keeps the internal log in a dedicated collection when the operator
asks for one, without changing where user data goes:
- New optional override, filer.options.metaLog.collection (and
.replication), read in NewFiler so both `weed filer` and
`weed server -filer` honour it. Default "" => exactly today's
behaviour (log follows the filer default), fully backward compatible.
- Resolution is a small helper: override first, then the filer default,
then a storage rule matched on the log path. Kept separate from the
user write path so the internal log targets itself.
- bucketCollection() is hardened the same way it already protects the
filer's default collection: a bucket that happens to resolve to the
redirected meta-log collection must not drop it on delete, because it
backs internal log volumes.
- Scaffold filer.toml documents the new knobs under [filer.options].
Related to the persisted deletion ledger branch (fix/persist-deletion-queue):
together they cut the two sources of post-flap junk in the default
collection — that PR stops orphaned user-chunk leak on filer crash,
this one stops the internal log from living in default at all. They are
independent: no file overlap, no functional dependency; either can merge
first. They are paired only in the narrative of cleaning up default.
Adds unit tests for the collection/replication resolution chain, the
viper keys, and the bucket-delete guard (run green under -race).
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.8-Flash-Next-ROCmFP4)
* filer: collect bucket chunks when its collection survives the delete
bucketCollection returning "" preserves the collection, but the bucket
path still skipped per-entry chunk collection and could skip listing the
children entirely, so a bucket sharing the meta-log (or any preserved)
collection left its object chunks orphaned with no entry pointing at
them. Only the wholesale drop of a deleted collection skips those now.
Note in filer.toml that the meta-log target should stay stable: chunks
written under an older collection are not migrated.
* filer: exercise the metaLog override wiring through NewFiler
The viper test only echoed back the keys it set, so a wrong key in
NewFiler would still pass. It now asserts the fields NewFiler fills
from those keys.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: tighten comments around the metaLog collection override
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: read GET/HEAD needles off the store lock, and only once
The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.
The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stream needle chunks without the store lock
StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.
The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.
The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stop a needle stream once its volume becomes unavailable
Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.
The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume server: split volume_copy into phases and type the delete-after-status gate
volume_copy was one ~400-line handler, and the rule that an existing
local replica is deleted only after the source's ReadVolumeFileStatus
succeeded was held by statement order alone. The keep_remote_data=true
that the pre-copy delete and the failed-copy rollback must share was
kept in sync by a comment pointing from one to the other.
The handler is now a ~60-line orchestrator over connect_to_copy_source,
SourceVolumeStatus::fetch, delete_existing_replica,
plan_copy_destination and a VolumeCopyJob whose run() drives
preallocate_dat, transfer_files, finish_copied_files and
mount_and_reply, with cleanup_failed_copy on error.
delete_existing_replica takes a &SourceVolumeStatus, which only fetch
can construct (private field in a child module), so the delete cannot
be called before the status RPC. Both deletes go through
delete_replica_keep_remote.
Pure refactor: call order, status codes and messages, cancellation
checks, throttling, progress reports and cleanup are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: find space for a VolumeCopy before deleting the replica it replaces
VolumeCopy deleted an existing local replica as soon as the source answered
ReadVolumeFileStatus and only then looked for a location with room for the
copy. With no usable location (disk full, low-disk, wrong disk type) the call
errored after the delete, leaving the node with neither the old replica nor
the new one.
Plan the destination first, as Go does: find_free_location_replacing credits
the location holding the replaced volume with that volume's slot, so a disk at
its volume limit that holds the replica still accepts the copy. Only then
delete the replica and write the .note (still after the delete, as in Go).
delete_existing_replica now takes the planned CopyDestination, so the delete
cannot precede the plan. find_free_location_predicate keeps its behaviour.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: describe the replace-credit test against the current VolumeCopy flow
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-01 23:13:17 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: restrict admin gRPC to local callers when no signing key
The S3 gateway's gRPC port (default 0.0.0.0:19000, always on) serves the
IAM cache and internal lifecycle admin services. checkAdminAuth was a
no-op when jwt.filer_signing.key was unset, so any reachable host could
PutIdentity an admin identity and take over the bucket data.
Without a shared key callers cannot be distinguished, so admin RPCs are
now limited to unix-socket, loopback, and the server's own interface
addresses. Remote filer-to-S3 propagation and lifecycle workers must set
jwt.filer_signing.key; the Bearer-token path is unchanged.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: fail closed on nil guard and refresh local addresses per call
Review feedback: a nil filerGuard bypassed all checks — treat it like a
missing key and require a local peer. The own-address set was cached
forever, so interfaces added later were rejected; enumerate per call
instead since admin RPCs are rare. Nil ctx is denied rather than panics.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: read the signing key once and bound interface enumeration
Review feedback: reading SigningKey twice could straddle a SIGHUP reload
— an old nonempty key skipped the local-peer check while the new empty
key verified the token. And enumerating interfaces per no-key call is
wasteful for co-located workers dialing the announced address; cache the
address set for 30s so new interfaces still become usable promptly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: enumerate interface addresses per no-key admin call
A cached address set keeps trusting an IP after it is removed from the
host and reassigned to another machine — that host would then hold
unauthenticated admin access for the cache TTL. Per-call enumeration only
runs for non-loopback TCP peers on the no-key path, which is low-volume
admin traffic, so the freshness is worth the syscall.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: refuse a tier move while compacting, and a commit once tiered
A tier move to remote and a vacuum compaction of the same volume could
interleave and leave the volume unreadable:
- A compaction committing while the upload ran swapped .dat/.idx under
the transfer, which reopens the .dat by path per part. The move then
published an object holding the old (or a mixed) layout against the
compacted .idx, and with keep_local_dat_file=false deleted the only
compacted .dat.
- A tier move finishing while the compaction copy ran (or between the
copy and the commit) let the commit swap in the compacted .idx while
the reload served the pre-compaction remote object through it.
The tier move now refuses to start while the volume is compacting, and
re-checks the compaction revision under the store write lock before it
records the remote file; on a mismatch it deletes the uploaded object
and fails with FailedPrecondition, leaving the volume local. Committing
a compaction on a volume that has a remote file is refused and its
.cpd/.cpx removed, since the reload would read the remote object
through the compacted index.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: abort a tier move whose volume was replaced or removed
The tier-up bookkeeping looked the volume up by id only and compared the
compaction revision. A delete and re-create of the same id during the upload
yields a fresh volume at the same revision, so the move recorded the old
volume's object on the new one and, without keep_local_dat_file, removed the
new .dat. An unmounted volume was skipped and the move reported success,
leaving the uploaded object referenced by nothing.
Capture the volume instance (its data-file access control Arc, as the scan
and read plans do) with the revision, and require both under the store write
lock. A replaced volume fails with FailedPrecondition, a missing one with
NotFound; either way nothing is recorded and the object is deleted after the
lock is released. Go fails in both cases because deleting or unmounting closes
the descriptor its copy reads.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: refuse to compact a volume tiered to remote storage
Committing a compaction of a tiered volume is refused, since the reload
would read the remote object through the compacted index. The compaction
itself still started: a tiered volume's data backend is the remote object
(the local .dat is dropped or deleted on tier-up), so an explicit vacuum
streamed the whole .dat out of remote storage into a .cpd that the commit
then discarded.
Refuse at the start of the compaction instead, before the .cpd is created,
at the point where Go's copy opens the local .dat. The truncated-index
test now uses a read-only local volume for its sorted index, since a
tiered one no longer reaches the copy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: match the ReadOnly(VolumeId) variant in write_volume_needles
#11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* ecbalancer: honour total-shards-per-rack cap in Place / PlaceDurabilityFirst
Worker auto-EC encode places via Topology.Place, which capped each shard
type independently (ceil(data/racks), ceil(parity/racks)). On an 8-rack
topology that permits 3 total shards on one rack, so losing two racks
strands 6/14 and a 10+4 volume becomes unreadable.
- tryPlace caps the total shards (data + parity) per rack in both modes,
whether or not ReplicaPlacement is set.
- rackTotalCap picks the smallest per-rack total the racks' real room
(free slots, bounded by the per-disk cap and node free slots, counting
shards already placed) can satisfy. On a uniform cluster it is
ceil(shards/racks); a nearly full rack raises it just enough that the
cap alone never fails an encode.
- PlaceDurabilityFirst gets a last rung that drops the rack cap
("rack-total-cap" in Relaxed), so it fails only when no disk has room.
PlaceStrict keeps the cap as a hard limit.
- chooseShardDest tries the next rack when the chosen one has no node
that fits, and room checks count the per-disk cap, so a rack whose
disks are all at the cap is no longer picked and then failed on
(pre-existing: 3-node rack + single-disk rack failed at shard 9).
- Docs no longer claim the cap guarantees surviving rack loss; the
placement error names the caps in effect; the encode warning no longer
says replica placement when other constraints were relaxed.
place_rack_cap_test.go covers 10+4 over 8 racks (max 2/rack, 3/rack on
master), a starved rack, nearly full racks, the preferred-tag tier, the
full-disk rack, and rackTotalCap directly.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* ecbalancer: size the rack total cap from room left under SameRackCount
The rack total cap counted each rack's free disk room, but attempts that
enforce ReplicaPlacement also stop a node at SameRackCount shards. With
SameRackCount=1, four one-node racks and four three-node racks got cap 2,
which fits only 12 of 14 shards: strict placement failed and
durability-first relaxed replica placement although 1 per small rack and
up to 3 per large rack fits.
Attempts that enforce ReplicaPlacement now use a cap sized from each
node's remaining SameRackCount allowance; attempts that relax it keep the
disk-room cap.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3: return 403, not 500, when an over-quota bucket refuses a write
Filer AssignVolume flattened ErrReadOnly into the free-text
AssignVolumeResponse.Error string, so S3 PutObject / PutObjectPart via
UploadReaderInChunks could not match it with errors.Is and fell through
to 500 InternalError: retryable, and it hides the quota.
Add FilerError READ_ONLY and AssignVolumeResponse.error_code, set it
alongside the unchanged error text, and rebuild the sentinel with
filer_pb.AssignVolumeResponseError. weed_server.ErrReadOnly now aliases
filer_pb.ErrReadOnly so errors.Is matches on both sides, and
mapChunkedUploadErrorToS3Error maps it to ErrAccessDenied. There is no
"read only" substring matching, so a volume server's "volume N is read
only" stays retryable.
Carrying the verdict as a response code rather than a gRPC status keeps
clients from treating it as a transport failure: the S3 gateway does not
fail over across filers and the Java client does not retry it.
Wrap per-chunk copy errors with %w so CopyObject keeps the sentinel, and
map UploadPartCopy chunk errors through mapCopyErrorToS3Error instead of
always returning 500.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* ci: re-run integration tests (PyPI download timeout)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: VolumeNeedleStatus reads remote EC shards and reports deleted needles like Go
For an EC volume the handler read only locally mounted shards, so a node
that did not hold the shard with the needle's bytes answered Internal
"ec shard N not available locally". Go's ReadEcShardNeedle fetches the
interval from a peer or reconstructs it. It also mapped every regular
volume read error, including a tombstone, to NotFound "needle not found",
which fs.verify treats as a missing needle; Go returns ErrorDeleted as a
plain error ("already deleted"), which fs.verify skips.
The EC branch now drops the store guard and uses the distributed EC read
the HTTP GET path uses. Errors map like Go: needle absent -> NotFound
"needle not found <decimal id>", tombstoned (regular or EC .ecx/.ecj) ->
Unknown "already deleted", anything else -> Unknown with the error text.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: tell EC deletions and vanished volumes apart in VolumeNeedleStatus
The distributed EC reader returned Ok(None) for an absent needle, a needle
a peer reported deleted, and a volume unmounted after the handler's own
existence check. VolumeNeedleStatus answered all three NotFound "needle not
found", which fs.verify -pruneEntries counts as lost data. A reported
deletion was also lost when an earlier interval failed.
The reader now says why it has no needle (EcMiss: NotFound, Deleted,
VolumeNotFound), classifying the local tombstone itself and letting a
reported deletion outrank other interval errors, as Go's ReadEcShardNeedle
does. VolumeNeedleStatus maps Deleted to Unknown "already deleted" and
VolumeNotFound to "volume not found", and drops its separate EC pre-check.
read_ec_shard_needle_distributed keeps its Ok(None) for every miss, so the
other callers are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: answer BatchDelete on EC needles as Go does
With skip_cookie_check, which every weed/ client sends, an EC needle that
was already deleted came back 404 "ec needle <fid> not found". Go's
DeleteEcShardNeedle gets ErrorDeleted from its read and BatchDelete
answers 304 with no error; the filer's deletion classifier only forgives
"already deleted" or an exact "not found", so it booked the repeat delete
as a permanent failure. The same mode also compared the fid cookie and
refused chunk manifests with 406, while Go never reads the needle before
those checks when skipping, so the filer's delete of a manifest chunk's
own fid failed permanently too.
The EC branch now reads with read_ec_shard_needle_or_miss and answers as
Go: skipping, a deletion is 304 and any other miss is 500 with Go's text;
checking, every miss is 404 with Go's text ("already deleted",
"locate in local ec volume: FindNeedleFromEcx: needle not found",
"ec shard <vid> not found"). The cookie and manifest checks run only when
the caller asked for the cookie check, which leaves the non-EC path as it
was.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: match the ReadOnly(VolumeId) variant in write_volume_needles
#11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
VolumeError::ReadOnly displayed "volume is read-only". Go's store and
volume say "volume %d is read only", and the filer's deletion classifier
requeues a failed delete only when the error contains "is read only".
Against a Rust volume server a BatchDelete on a read-only volume (tier
move, maintenance) was booked as a permanent failure and the chunk was
never deleted.
ReadOnly now carries the volume id and displays Go's text. The text
reaches clients through BatchDelete results, the HTTP write and delete
error bodies, and gRPC statuses; the gRPC code (FailedPrecondition) and
the HTTP/BatchDelete status codes are unchanged.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume: split the write path into reusable steps
do_write_request ran its pre-append checks, the append, the sync
rollback, the index publish and the post-write bookkeeping inline, so a
batched write could only reuse it one needle at a time. Pull the steps
out (check_writable, prepare_write, undo_unsynced_append, publish_write,
finish_write) and the store's volume lookup plus disk-space check
(writable_volume_mut). do_write_request composes them in the same order
with the same early returns; no behaviour change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: group-commit fsync writes in the write queue
The write queue holds one store lock for a batch of up to 128 needles
but wrote them one at a time, so every fsync needle paid its own .dat
sync and its own .idx sync: 2N syncs per batch.
Add Volume::write_needles_grouped, after Go's processBatch. A volume's
entries are split into runs of distinct needle ids (a repeated id starts
a new run, so its dedup and cookie checks see the earlier write). A run
with a durable entry appends everything with append_at_ns chained
through a local, syncs the .dat once, and only then publishes the
entries and syncs the .idx once. A failed .dat sync truncates the .dat
back to the run start (marking the volume unavailable if that fails),
leaves last_append_at_ns and last_modified untouched, and fails every
entry of the run. Runs with no durable entry go through the unchanged
per-needle path. Store::write_volume_needles is the queue's entry point;
the handlers' non-queue path is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: fail closed when a failed append's rollback fails
append_needle discarded the truncate-back result, so a partial write
that could not be rolled back left unindexed bytes on the .dat while
the volume stayed writable; the next append would bury them mid-file,
past the load-time tail check. Route the rollback through
undo_unsynced_append, which marks the volume unavailable when the
truncate fails, so nothing more is appended over an unverified tail.
* volume server: keep a grouped run's I/O error streak from later appends
A synced run stages every append before any entry finishes, so the
success reset in finish_write ran after the failed appends queued
behind the last write to land and erased their media-error streak.
Sent one at a time, those errors would have counted and quarantined
the volume. Skip the reset when an append after the last landed write
added to the streak.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: replay a grouped run's I/O error streak in queue order
Skipping the run's success reset whenever an append after the last
landed write failed kept the errors from before that write as well,
so a run like [EIO, EIO, landed, EIO] reached the quarantine count
that the same writes one at a time (one error) do not. Mark the
streak where each entry is staged and record the run's success at
the last landed write's mark: errors before it are cleared, the ones
after it still count.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: replay a run's I/O error streak in one atomic step
Reads record their outcomes on the tracker without the volume's write
lock, so record_success_at's separate load and store could drop an
error a read counted in between, or restore errors a read had just
cleared. Keep the count and the clear counter in one atomic word and
apply the replay with a single fetch_update.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
In readMode=proxy, proxy_request copied the target's response headers
with HeaderMap::insert, so a header the target sent more than once
(several Set-Cookie, Vary, Link, ...) reached the client with only its
last value. Go's proxyReqToTargetServer adds every value with
w.Header().Add. Append instead of insert; the Server header is still
dropped and status and body handling are unchanged.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: collect EC heartbeats and adjust volume max off the runtime
The volume pass moved to the blocking pool, but the heartbeat task still
called collect_ec_heartbeat and the following EC shard snapshot, and
Store::maybe_adjust_volume_max, directly on its tokio worker.
maybe_adjust_volume_max runs statvfs on every auto-sized disk and stats the .dat
of every writable volume under the store read lock. All of them block the
worker on the node-wide RwLock<Store> whenever a writer holds it or is
queued, and every task sharing that worker stalls with it.
Run the adjustment, on the pulse and after the master changes volume
options, and the EC tick's heartbeat plus shard snapshot through
off_runtime, like the volume pass. apply_master_volume_options now only
reports whether the options changed; the loop adjusts off the runtime.
What is collected and sent, and in what order, is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: keep EC shard deltas out of the volume heartbeat's snapshot
EC shard deltas are the diff between the store's shards and last_ec_shards,
taken when volume_state_notify fires. But the volume tick and the
options-changed heartbeat re-took last_ec_shards from the store too, and a
volume heartbeat carries no shard list: a mount or unmount that landed while
the notify was pending or the volume pass was collecting was absorbed into
the baseline and never sent. The master only learned of it at the next EC
tick, 17 pulses later. The EC tick likewise built its heartbeat and its
baseline under two separate store guards, so a mount between them was lost
the same way.
A volume heartbeat now only takes out of the baseline the expired EC shards
it reports deleted itself, so the next delta does not repeat them. The EC
tick, and the initial EC heartbeat, build the full list and the baseline
under one read guard, still on the blocking pool.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: take the heartbeat's remaining store reads off the runtime
The heartbeat's volume and EC passes and the volume max adjustment already
run on the blocking pool, but several reads of the node-wide RwLock<Store>
were still taken directly on the heartbeat's tokio worker: the digest
report reset before the first heartbeat, the duplicate-UUID directory
lookup and the volume options a master response carries, the EC shard list
a state notification is diffed against, and the deregistration heartbeat
sent on stop and shutdown. The lock is writer-preferring, so with a writer
holding or queued for it each of these parks the worker, and every task
sharing that worker stalls with it.
Run each through off_runtime, which now takes a closure so a pass can carry
what it needs from the master's response. The notify branch's volume
snapshot and EC read become one blocking pass, still under two guards in
the same order. What is collected and sent, and in what order, is
unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume server: refuse a tier move while compacting, and a commit once tiered
A tier move to remote and a vacuum compaction of the same volume could
interleave and leave the volume unreadable:
- A compaction committing while the upload ran swapped .dat/.idx under
the transfer, which reopens the .dat by path per part. The move then
published an object holding the old (or a mixed) layout against the
compacted .idx, and with keep_local_dat_file=false deleted the only
compacted .dat.
- A tier move finishing while the compaction copy ran (or between the
copy and the commit) let the commit swap in the compacted .idx while
the reload served the pre-compaction remote object through it.
The tier move now refuses to start while the volume is compacting, and
re-checks the compaction revision under the store write lock before it
records the remote file; on a mismatch it deletes the uploaded object
and fails with FailedPrecondition, leaving the volume local. Committing
a compaction on a volume that has a remote file is refused and its
.cpd/.cpx removed, since the reload would read the remote object
through the compacted index.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: abort a tier move whose volume was replaced or removed
The tier-up bookkeeping looked the volume up by id only and compared the
compaction revision. A delete and re-create of the same id during the upload
yields a fresh volume at the same revision, so the move recorded the old
volume's object on the new one and, without keep_local_dat_file, removed the
new .dat. An unmounted volume was skipped and the move reported success,
leaving the uploaded object referenced by nothing.
Capture the volume instance (its data-file access control Arc, as the scan
and read plans do) with the revision, and require both under the store write
lock. A replaced volume fails with FailedPrecondition, a missing one with
NotFound; either way nothing is recorded and the object is deleted after the
lock is released. Go fails in both cases because deleting or unmounting closes
the descriptor its copy reads.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
EcVolumeShard::read_at and the scrub plan's EcLocalShard::read_at were a
single pread/seek_read. That may legally return fewer bytes than asked
mid-file (FUSE/NFS/CIFS mounts, a signal, very large requests), and an
Interrupted error was not retried. Callers treat a short count as end of
file or corruption: verify_ec_shards compared a zero tail and reported a
parity mismatch, local scrub reported a broken shard, VolumeEcShardRead
ended the stream early, and decode/rebuild/local needle reads failed.
Add storage::io::read_full_at, which loops until the buffer is full or a
read returns 0 and retries Interrupted, so a short count means EOF. Route
both shard read_at methods through it, replace the encoder's private
read_at_most with it, and reuse it for the Windows read_exact_at loop.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: ec.decode reads the .ecx from the index dir it was copied to
VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so
with a split data/index dir the decode target has no .ecx beside its
shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but
built the .idx from the data dir, failing with NotFound after the .dat
was already published. It now reads .ecx/.ecj from where the EC volume
opened them and writes the .idx beside the .dat, where Go leaves it.
The live-entry check and the .dat size also ignored deletions recorded
only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first:
a fully deleted volume was decoded instead of reported as having no live
entries, and deleted tail needles were copied into the .dat. Both now
treat journaled ids as deleted, without rewriting the sealed .ecx.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: ec.decode keeps the decoded volume writable and reads every .ecj
The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after
the .dat was cut short before it, so the mount saw a row past EOF and
marked the decoded volume read-only. Rows of deleted needles the .dat no
longer holds are now dropped, and each journaled needle still in the .dat
gets one tombstone instead of one per journal entry.
VolumeEcShardsCopy appends journals collected from other holders into
the idx dir, but the decode read only the .ecj beside the .ecx, which
sits in the data dir when this server generated the shards. It now
reads both, once, in bounded chunks via the loader EcVolume uses.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: test ec.decode drops a sealed .ecx tail tombstone
Covers the other half of the rule added in the previous commit: a tail
needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from
the .dat, and its row must not reach the rebuilt .idx either.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: VolumeNeedleStatus reads remote EC shards and reports deleted needles like Go
For an EC volume the handler read only locally mounted shards, so a node
that did not hold the shard with the needle's bytes answered Internal
"ec shard N not available locally". Go's ReadEcShardNeedle fetches the
interval from a peer or reconstructs it. It also mapped every regular
volume read error, including a tombstone, to NotFound "needle not found",
which fs.verify treats as a missing needle; Go returns ErrorDeleted as a
plain error ("already deleted"), which fs.verify skips.
The EC branch now drops the store guard and uses the distributed EC read
the HTTP GET path uses. Errors map like Go: needle absent -> NotFound
"needle not found <decimal id>", tombstoned (regular or EC .ecx/.ecj) ->
Unknown "already deleted", anything else -> Unknown with the error text.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: tell EC deletions and vanished volumes apart in VolumeNeedleStatus
The distributed EC reader returned Ok(None) for an absent needle, a needle
a peer reported deleted, and a volume unmounted after the handler's own
existence check. VolumeNeedleStatus answered all three NotFound "needle not
found", which fs.verify -pruneEntries counts as lost data. A reported
deletion was also lost when an earlier interval failed.
The reader now says why it has no needle (EcMiss: NotFound, Deleted,
VolumeNotFound), classifying the local tombstone itself and letting a
reported deletion outrank other interval errors, as Go's ReadEcShardNeedle
does. VolumeNeedleStatus maps Deleted to Unknown "already deleted" and
VolumeNotFound to "volume not found", and drops its separate EC pre-check.
read_ec_shard_needle_distributed keeps its Ok(None) for every miss, so the
other callers are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The volume pass moved to the blocking pool, but the heartbeat task still
called collect_ec_heartbeat and the following EC shard snapshot, and
Store::maybe_adjust_volume_max, directly on its tokio worker.
maybe_adjust_volume_max runs statvfs on every auto-sized disk and stats the .dat
of every writable volume under the store read lock. All of them block the
worker on the node-wide RwLock<Store> whenever a writer holds it or is
queued, and every task sharing that worker stalls with it.
Run the adjustment, on the pulse and after the master changes volume
options, and the EC tick's heartbeat plus shard snapshot through
off_runtime, like the volume pass. apply_master_volume_options now only
reports whether the options changed; the loop adjusts off the runtime.
What is collected and sent, and in what order, is unchanged.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mount_and_reply ignored the Result of store.unmount_volume when a copied
replica failed record count validation, tripping unused_must_use. The Err
branch is unreachable there: the volume was mounted under the same store
write guard, mount_volume refuses an already loaded vid so it is a fresh
Volume with is_compacting false, and a compaction claim needs &mut Volume,
i.e. the store lock. Ignore the result explicitly with a one-line reason.
Store::delete_expired_ec_volumes and Store::remove_ec_volume are called
only from test modules (the heartbeat uses the split
find_expired_ec_volumes / remove_expired_ec_volumes halves), so mark them
#[cfg(test)].
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GetObjectAttributes accepted the Checksum attribute but never filled it
in, as its comment said SeaweedFS did not store S3 checksums. PutObject
and CompleteMultipartUpload store them now, and HeadObject returns them.
Fill in Checksum from the same entry fields, with the ChecksumType and
ChecksumCRC64NVME members the response did not have.
Also run ceph/s3-tests' test_get_checksum_object_attributes in CI.
* s3api: do not store aws-chunked in an object's Content-Encoding
aws-chunked in Content-Encoding names the SigV4 streaming framing of
the request body, which the gateway decodes on upload. PutObject and
CreateMultipartUpload stored the header as sent, so an object uploaded
with "gzip, aws-chunked" was served with that Content-Encoding, and one
uploaded with "aws-chunked" alone was served as aws-chunked. S3 drops
aws-chunked and keeps the other encodings.
Also run ceph/s3-tests' test_object_content_encoding_aws_chunked in CI.
* s3api: read every Content-Encoding field, and drop aws-chunked on copy
A client can send aws-chunked and the object's own encoding as separate
Content-Encoding fields. Only the first was read, so "aws-chunked"
followed by "gzip" left the object without its gzip. Combine all the
fields before dropping aws-chunked. CopyObject with the REPLACE
directive stored the requested Content-Encoding as sent: drop
aws-chunked there too.
A replica's read-only flag in the volume layout only moved on registration
and on volume.mark. A change that arrived in the regular heartbeat updated
the node's record, which the writable list follows, but not the layout
flag, which the vacuum sweep reads. So the sweep kept trying volumes on a
disk that had gone read-only while the server ran, and after a restart it
skipped volumes that had since become writable again until the next
restart (issue #11516).
Apply the reported state to the flag for every changed volume. Only the
flag: the writable list stays with EnsureCorrectWritables and its
capacity guards.
* shell: say which read-only volumes volume.vacuum leaves alone
volume.vacuum without -volumeId runs the same sweep as the automatic
vacuum, which skips read-only volumes, and the master's response carries
no result. An operator whose disk filled up runs the command, sees it
return, and watches nothing change (issue #11516).
Before issuing the request, list the read-only volumes whose garbage is at
or above the threshold and point at -volumeId, which is the explicit path
PR #9861 opened for them. The help text says the same.
* shell: volume.vacuum hint survives a failed listing and looks at every replica
Review follow-ups: a failed topology listing no longer stops a sweep
without -volumeId, it only drops the hint; a volume counts as read-only
when any replica is, with the garbage ratio taken from the replica that
reports the most, which is what the sweep itself does; a converted index
that reports deletes without sizes is listed rather than hidden; and the
threshold is printed as given instead of rounded to two decimals.
* shell: do not guess a garbage ratio for a converted index
The master cannot compute one for a volume that reports deletes without
their sizes, and a guess of 1 would send the operator to -volumeId for a
volume the server may decline at that threshold. Leave it out and say so.
* s3/iam: manage roles through the IAM API, with an opt-in persistent role store
Roles could only come from the IAM config file: the S3 server pinned the
role store to memory and the embedded IAM API had no role actions, so a
role could not be created, retrusted or revoked without editing the file
and restarting every gateway.
Role store
- Read the `roleStore` key (the IAMConfig field already existed). With an
IAM config file the default stays memory; with none it is the filer, as
for OIDC providers, so zero-config clusters keep runtime-created roles.
- Roles from the IAM config file never go into a persistent role store,
which outlives the file and may be shared by S3 servers with different
files. They are served from memory beneath the store, as OIDC providers
are: a stored role of the same name takes precedence, and deleting it
restores the file's. A config-file role cannot be changed or deleted
through the API (UnmodifiableEntity), and removing one from the file
removes it at the next start. An in-memory store holds them as records,
as before. They have no creation time, so CreateDate is omitted rather
than reporting when this server started. SetRoleStore installs a store
the same way, so a store set after startup keeps the config-file roles,
as SetOIDCProviderStore does for providers.
- Watch /etc/iam/roles and drop the cached role definitions on change. The
cached filer store otherwise serves a peer's stale role for up to its 5m
TTL, which keeps a revoked trust policy in force on the other gateways.
- Role stores wrap ErrRoleNotFound for a missing role; the filer store
used to report any failed lookup as "role not found". CreateRole proceeds
only on a confirmed absence, so an unreadable store cannot let it write
over an existing role.
IAM actions
- CreateRole, GetRole, ListRoles, DeleteRole, UpdateAssumeRolePolicy,
AttachRolePolicy, DetachRolePolicy, ListAttachedRolePolicies. The reads
are allowed in read-only mode.
- A role defined in the config file is reloaded from it at every start, so
changing or deleting it through the API is refused (UnmodifiableEntity)
rather than silently reverted.
- DeleteRole with policies attached is refused (DeleteConflict), as on AWS.
- Role names follow AWS's rules ([\w+=,.@-]{1,64}); a role is stored as
<name>.json in the filer, so this also keeps a name from leaving the role
store's directory. At most 10 managed policies per role (AWS's default
quota; MaxManagedPoliciesPerUser is 10 too), LimitExceeded beyond.
- DeletePolicy is refused (DeleteConflict) while a role attaches the
policy, as it already is for users and groups: roles attach policies by
name, so a policy created later under the deleted one's name would
otherwise take effect on the role.
- Role paths other than "/" and role tags are not stored, so they are
refused rather than dropped.
Role IDs and sessions
- Roles get a unique RoleId when first stored (random, AWS AROA form),
kept across updates; a config-file role gets a stable ID derived from its
name, since it is created again at every start.
- Sessions issued through AssumeRoleWithWebIdentity, AssumeRoleWithCredentials
and AssumeRole carry the role's ID (claim "rid"), and a request under a role
whose current ID differs is denied. Resolving a session's policies by role
name let a session outlive its role: once a role was deleted, a role later
created under the same name — with a different trust policy and different
policies — revived every unexpired session of the old one with the new
role's permissions. Sessions issued before this change carry no ID and are
unaffected until they expire.
Integration test (test/s3/iam, run with `make start-services`):
TestWebIdentityWithProviderAndRoleManagedThroughIAMAPI configures an OIDC
provider, a managed policy and a role entirely through the IAM API against a
JWKS served by the test, then checks the trusted subject gets credentials
scoped to the attached policy; another subject, a token signed by another
key, an unsigned token and a token for another audience are refused; and UpdateAssumeRolePolicy moves the
trust at once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: bind every role session to its role, and change roles atomically
Review follow-ups.
Session binding
- The role-ID check ran only when a session carried no policy names, and
AssumeRole embeds the role's attached policies, so those sessions kept
their permissions after the role was deleted or recreated. The check
now runs for every session carrying a role ID, before policy selection.
- A named role that cannot be resolved at issuance gets no session,
instead of one with no role ID (which nothing binds).
- A config-file role's ID is derived from its name and trust policy, not
the name alone: a different role put in the file under the same name
gets a new ID, while an unchanged role keeps its sessions across restarts.
Role writes
- RoleStore gains UpdateRole, a read-modify-write that lands only if the
role is unchanged since the read, and otherwise re-reads and retries. The
filer store uses the filer's write conditions (IF_NOT_EXISTS for a new
role, IF_ENTRY_EQUAL otherwise). CreateRole, UpdateAssumeRolePolicy and
Attach/DetachRolePolicy all go through it, so two gateways no longer
overwrite each other's changes, a change racing a delete no longer
writes the role back, and of two concurrent creates one gets
EntityAlreadyExists.
- The filer store's ListRoles pages past 1,000 entries and fails on a
broken stream instead of returning what arrived, so DeletePolicy's
attachment check sees every role. ListRoles skips a role deleted between
listing and reading it.
- CreateRole validates first; a failed write is ServiceFailure, not
InvalidInput. Any Tags.* parameter is refused, not only the first key.
- ExecuteAction's skipPersist covers the S3ApiConfiguration only; the
comment now says so. Role and OIDC provider actions write their own stores.
Each fix has a test that fails without it. Against a real filer with two
gateways, concurrent AttachRolePolicy calls lost 1-4 of 8 attachments per
run before this change and none after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: one role snapshot per decision; DeleteRole is atomic; watch a custom role store path
Review follow-ups.
- Authorization evaluates the policies of the role definition the session's
binding was checked against, instead of reading the role again: a role
replaced in between cannot lend a session its policies.
- AssumeRole and AssumeRoleWithLDAPIdentity issue the session from the
definition whose trust admits the caller (IAMManager.ResolveRoleForPrincipal),
and take its ID, duration cap and embedded policies from that same
definition. A role replaced after the caller's trust check by one that does
not trust the caller now yields AccessDenied, not a session bound to the
replacement.
- A RoleUpdate that returns nil deletes the role, on the same condition as a
write: the filer store deletes with ObjectTransaction on IF_ENTRY_EQUAL,
routed and locked like the conditional CreateEntry. DeleteRole decides
against the role it deletes, so a policy attached meanwhile on another
server is a DeleteConflict, and a delete never removes a role written
after its check.
- S3 servers watch the role store's configured basePath, not only
/etc/iam/roles, so a custom path also drops peers' cached roles on change.
Each has a test that fails without it. Live against a real filer: DeleteRole
refuses while a policy is attached and removes the entry once detached; all
test/s3/iam CI stages pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: state which roles DeletePolicy's attachment check can see
RolesAttachingPolicy sees the stored roles and this server's config-file
roles. A role defined only in another server's IAM config file is invisible
to it, so a config-file role that attaches a managed policy is protected
only on the servers whose file defines it. The doc comment now says so and
how to avoid it: keep such roles in every server's file, or attach only
config-file policies to config-file roles.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: note that a role store set after startup is not watched for peer changes
S3 servers build their metadata watch list once, at startup, from the role
store installed then. SetRoleStore's doc now says that a filer-backed store
installed later with a different basePath is not watched, so peers' changes
to it reach this server's cached roles only when the cache expires.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: retry a failed OIDC provider refresh until the store answers
RefreshOIDCProvidersFromStore reports a failure and nothing retries it. Its
callers can't: a metadata-subscription event reports each change once, so a
refresh that found the filer unreachable on it (the filer restarting, say)
left a peer's new provider untrusted, or a deleted one trusted, until some
unrelated later change. The refresh after a local IAM API mutation has the
same shape. Only the startup load retried.
A failed refresh now retries in the background with the startup load's
backoff until the store answers. At most one retry runs, however many
refreshes fail meanwhile, and installing another store cancels it. The
startup load uses the same path instead of its own.
Seen on a SeaweedFS operator cluster whose filer restarted while an
S3OIDCProvider was created: the gateway logged "OIDC provider refresh after
/etc/iam/oidc-providers change failed: ... fail to dial". The operator's
periodic re-apply happened to recover it; an IAM API client would not.
* s3/iam: never retry or apply a superseded OIDC provider store, and never drop a failure during a retry
Review of the retry (#11521) found two ways to lose the state it protects.
A refresh of store A that failed as store B was installed could start a
retry for A after B's install had cancelled retries. Nothing cancelled it,
and when A answered it replaced B's providers in STS. The installed store
now changes under the retry lock, a store that is no longer current gets no
retry, and a snapshot of a replaced store is never handed to STS, even when
the refresh listed it just before the swap.
A refresh that failed while a retry ran was dropped by the at-most-one
guard, though the retry might already have listed an older snapshot, so the
change the failed refresh would have loaded stayed unloaded. The retry now
runs once more after its success when a failure arrived meanwhile.
Each has a test that fails without its guard.
2026-09-30 17:33:40 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage/topology: keep disk-full read-only volumes vacuumable
The vacuum sweep skipped every read-only replica, so a volume that went
read-only because its disk filled could never reclaim its garbage — the
exact situation compaction exists for. The volume server now reports
disk_space_low in VacuumVolumeCheckResponse, and the sweep skips a
read-only replica only when the flag is clear. An explicit volumeId
vacuum is unaffected: it already bypassed the read-only rule.
The field takes number 4: 2 and 3 are downstream-allocated for tombstone
retention, keeping the wire merge clean.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: measure vacuum free space against live bytes
The pre-compaction space check required the current .dat + .idx size
free, which includes the garbage being reclaimed — on a nearly full disk
that estimate can never fit, so the volume stayed garbage-bound forever.
Measure against the estimated compacted output instead: superblock plus
live index entries plus live content bytes, with the existing ten
percent buffer unchanged. Mirrors the same check in the Rust volume
server.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: count per-needle framing in the compacted-size estimate
The live-bytes estimate covered each live needle's content and index
entry but not its .dat framing (header, checksum, timestamp, padding —
~32 bytes on version 3). For small-needle volumes that is more than the
10% headroom, so a disk with space between the estimate and the real
output still ran out mid-compaction. Rust side mirrors the same formula.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: report disk_space_low only when it is the sole read-only cause
Review feedback (ihnokim, greptile, devin): a volume read-only for low
disk space AND an operator mark or I/O quarantine was still eligible for
the automatic sweep, rewriting a copy meant to stay protected. The flag
now reports only the benign sole-cause case in both servers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: fail closed when the read-only lookup misses in the sweep
A heartbeat can drop the volume from the DataNode cache between the
location-list copy and VacuumVolumeCheck; a lookup error previously
skipped the read-only check entirely. Review feedback (coderabbit).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CopyObject responses carried LastModified values such as
"2026-09-29T20:30:04.56Z": trailing zeros of the fractional seconds were
trimmed, and a whole-second value had no fraction at all. AWS S3 always
writes exactly three digits ("...04.560Z"), and clients that parse with a
fixed-width pattern reject anything else. minio-java 8.6.0
(yyyy-MM-dd'T'HH:mm:ss.SSS'Z') throws DateTimeParseException, so roughly
one CopyObject in ten fails on the client even though the copy succeeded.
Two causes:
- xsdDateTime marshalled with "2006-01-02T15:04:05.999999999", which
drops trailing zeros. It now writes UTC with ".000Z".
- CopyObjectResult.MarshalXML had a pointer receiver, but the handlers
pass the result by value, so encoding/xml never called it and fell back
to time.Time's RFC 3339 encoding. It now has a value receiver.
CopyPartResult had no custom marshaller at all; it now uses xsdDateTime.
Follow-up to #8394 / #8398, which truncated these timestamps to
milliseconds but kept the trimmed format.
* filer: identify remote-mounted entries safe to drop under disk pressure
ListEvictableRemoteEntries walks every mounted directory directly on the
filer store (no lazy remote listing) and returns entries that hold local
chunks fully synchronized with remote, ordered oldest-cached first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: evict remote-cached chunks oldest-first and vacuum the garbage
uncacheRemoteEntry applies the same transition remote.uncache does -
cleared chunks plus a reset LastLocalSyncTsNs under the entry path lock -
and evictRemoteCachedEntries serializes passes over all mounts until a
byte target is met. Aged victims are preferred; a second pass accepts any
synchronized cached entry when aged ones cannot cover the request, since
a failed read is worse than a dropped hot object.
Cleared chunks only become disk space after compaction, so
reclaimRemoteCacheSpace pairs each pass with a rate-limited VacuumVolume
call that also picks up orphaned partial fills.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: trigger remote cache eviction under storage pressure
A periodic check (30s) reads disk usage from master topology and evicts
remote-mounted cached chunks once any disk crosses
-filer.remoteCacheEvictThreshold (default 0.9; 0 disables), with a vacuum
pass to reclaim the tombstoned needles.
The cold-read cache path also kicks the same reclaim when a fill fails on
exhausted volumes - the request still falls back to streaming from the
remote, but the cache stops being permanently wedged full.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: flush deletion queue before remote cache vacuum
Vacuum ran immediately after eviction while evicted file IDs still sat
in the asynchronous deletion queue, so compaction saw no garbage and the
cache stayed wedged. Flush the queue synchronously first and shorten the
vacuum cooldown so sustained pressure does not wait five minutes between
reclaim passes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover remote cache eviction under capacity pressure
Unit tests pin the eligibility filter and oldest-first ordering; the
integration test runs a constrained two-node setup that saturates the
cache, verifies the oldest synced entry is evicted and vacuumed, and
that a later read re-caches it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: coalesce remote cache reclaim passes
A failed cache fill used to queue behind any in-flight eviction,
stacking full mount traversals during a write-failure storm. Skip the
pass when one is already running; the caller falls back to streaming
from remote regardless.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: stop the remote cache janitor on shutdown
The eviction ticker kept running after Shutdown closed the metadata
store and could traverse a closed store. Give the janitor a context
cancelled from Shutdown and propagate it into its master RPCs and
traversals.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: vacuum only tombstoned volumes and retry deferred passes
VacuumVolume with no volume id swept every collection, compacting
volumes unrelated to the cache fill that failed. Now the reclaim path
collects the vids of file ids actually flushed from the deletion queue
and compacts only those. Vids that land inside the vacuum cooldown stay
in a pending set the janitor retries on each tick, so chunks evicted
just after a sweep are not stranded until the next pressure event.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: count only pressured disks when evicting remote cache
The janitor measured the largest excess on one disk but let bytes on
healthy disks satisfy the reclaim target. Split the topology disk view
per physical disk and count only chunk bytes whose volumes sit on an
over-threshold disk; entries contributing nothing there are skipped.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: compare remote cache sync time at nanosecond precision
Second-precision mtime comparisons let a local write in the same second
as the last sync still qualify as evictable, discarding unsynced
changes. Compare LastLocalSyncTsNs against full-precision mtime
(mtime_ns round-trips through the entry codec), and apply the same fix
to remote.uncache's inline check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: invalidate remote sync stamp on local content change
A local overwrite that keeps the remote entry's LastLocalSyncTsNs looks
evictable even though the remote copy no longer matches, and some write
paths stamp mtime at second precision so a timestamp comparison cannot
catch it. UpdateEntry now clears the stamp when chunks change without a
fresh stamp, leaving replicated updates authoritative.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound remote cache master rpcs and vacuum all evicted garbage
VolumeList and VacuumVolume now run under a 30s context so a stalled
master cannot wedge the eviction janitor. The targeted vacuum drops the
garbage threshold so volumes with under 10% deleted bytes still compact.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: tolerate straggler fills in remote cache eviction test
Detached fills from the concurrent wave keep racing the final checks:
live chunks legitimately fill both volumes, and a re-cached object can
be evicted again before its commit is observed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: start remote cache eviction loop after filer init
The janitor's first tick dereferences fs.filer; starting the goroutine
before NewFiler assigns it could panic when startup exceeds an interval.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep remote cache vacuum intent across retries
Evicted entries now record their chunk volumes for vacuum directly, so
the intent survives whoever consumes the shared deletion queue first.
A pending volume keeps several vacuum attempts so tombstones that land
late are still compacted, and the janitor retries pending volumes under
the reclaim mutex instead of flushing unrelated deletes every tick.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound each remote cache vacuum request independently
A shared 30s deadline across pending volumes let one slow compaction
cancel the rest. Each VacuumVolume now gets its own context, and pending
volumes keep more attempts since the master reports request acceptance
rather than compaction.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: tighten remote cache reclamation bound
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: treat chunk timestamp changes as content changes
chunksEqual now also compares ModifiedTsNs so an update that rewrites a
chunk record still invalidates the remote sync stamp.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: run remote cache queue flush under the reclaim context
BatchDelete for flushed file ids now uses the caller's context instead of
context.Background(), so a reclaim pass bounded by shutdown or timeout
stops its deletes too. Other callers keep their existing behavior.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: scope remote cache vacuum to evicted volumes
The flush no longer feeds the shared deletion queue's ids into the
pending set — only evicted chunks' volumes are tracked, so ordinary
deletions no longer pick up repeated vacuum attempts. The flush also
runs under a shutdown-immune bounded context and is skipped when no
volume is pending.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: retry remote cache vacuums even after unmount
Pending volumes were only retried while a remote mount existed; removing
the last mount skipped every later pass and left evicted bytes allocated.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: reclaim partial cache fills that run out of capacity
A fill that fails midway queues its written chunks for deletion, but
when no entries remain evictable the reclaim pass found no pending
volumes and skipped the flush and vacuum entirely, leaving the partial
garbage to the slow periodic vacuum while the disk stayed full. Mark
the failed fill's chunk volumes pending so the pass tombstones and
compacts them even when nothing was evicted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add ppc64le and s390x to the normal and large_disk release image
matrices, next to arm/v7 and 386, and to the manual container_latest
fallback so that rebuilding `latest` does not drop them from the
manifest.
The Go binary is cross-compiled without cgo on the build host, and
Dockerfile.go_build already falls back to empty Rust volume server and
worker placeholders on platforms without pre-built Rust binaries. Only
the small final-stage RUN steps run emulated, so these two platforms use
QEMU on ubuntu-latest like arm/v7.
Refs #5558
2026-09-29 21:04:19 +08:00
Khris RichardsonGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Claude Opus 5.5Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3/iam: persist IAM-managed OIDC providers in the filer, and trust them after a restart
The S3 server's IAM config loader never read the documented
`oidcProviderStore` key, so the OIDC provider store was always in memory:
a provider created with CreateOpenIDConnectProvider lived in one gateway's
process, was lost on restart, and was never seen by peers. The
/etc/iam/oidc-providers metadata subscription refreshed from that empty
in-memory store.
- Read `oidcProviderStore` and pass it to the IAM manager. With an IAM
config file the default stays memory. With no config file (zero-config
IAM, as `weed filer -s3` and operator-managed clusters run) it defaults
to the filer: there is nothing static to shadow, and providers created at
runtime otherwise vanish on restart.
- With a store that outlives the process, load the STS runtime view from it
at startup, so providers created on an earlier boot or on a peer are
trusted without waiting for the next mutation.
- If the store cannot be read at startup (a filer not up yet), the load is
retried in the background with backoff until it succeeds: the metadata
subscription reports only later changes, so providers already stored would
otherwise stay unknown to STS until one of them changed.
- Mark records mirrored from STS.Providers as `source: static-config`, and
at startup delete such records whose provider has left the config, so
removing a provider from the config file still revokes it. Records created
through the IAM API are never pruned.
- The filer store reported every failed lookup, an unreachable filer
included, as ErrOIDCProviderNotFound, which CreateOIDCProvider reads as
"free to create". Only a confirmed absence is now not-found.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: keep config-file OIDC providers out of a persistent store
Review of the previous commit found that mirroring the IAM config file's
providers into a persistent store, and pruning them when they leave the
file, breaks as soon as S3 servers share a filer:
- a server prunes stored config-file providers its own file does not list,
including ones a peer's file still defines (a zero-config server prunes
them all);
- mirroring overwrites an API-created provider with the same ARN and marks
it config-owned, so a later prune deletes it;
- a failed mirror write or a failed prune leaves a stale record trusted;
- a mirrored record is loaded into STS at startup as an IAM-managed provider
and shadows the config-file provider, dropping the settings a record does
not carry (jwksUri, roleMapping, policyClaim, ...).
A persistent store now never receives the config file's providers. STS keeps
serving them from its static configuration, as it always has; the IAM API
lists and returns them from memory, refuses to change or delete them
(UnmodifiableEntity; change them in the file) and to create another provider
with their ARN (EntityAlreadyExists). The store holds only providers created
through the IAM API, and those are what startup loads into STS. There is
nothing to prune, so the source marker is gone. An in-memory store keeps its
behaviour: the config file's providers are records in it, as before.
buildOIDCProviderFromRecord also carries PolicyClaim and
AllowedPrincipalTagKeys now; they were dropped whenever an API-created
provider was loaded into STS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: send UnmodifiableEntity as a 400, not an internal error
The IAM API's error writer had no case for UnmodifiableEntity, which the
previous commit returns for a change to a config-file provider, so it went
out as a 500 ServiceFailure that clients retry. AWS sends it as a 400.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: document stored-over-config precedence, drop invented CreateDate, cancel superseded retries
Follow-ups from review of b881982d2:
- A provider stored under the same ARN as a config-file provider takes
precedence in the IAM API, matching STS, which already prefers
IAM-managed providers so that an API call can shadow a bootstrap entry.
Deleting the stored provider brings the config-file one back. This was
already the behaviour; it is now documented and tested.
- A config-file provider no longer reports its server's start time as
CreateDate, which changed on every restart; GetOpenIDConnectProvider now
omits the date for it. An in-memory store still stamps its copies at load,
as before.
- The startup retry runs under a cancellable context, is cancelled when
another store is installed, and retries the store it was started for
rather than reading the manager's field, so replacing the store neither
leaves the old retry running nor races with it (go test -race).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: serialize OIDC provider refreshes so an older snapshot cannot restore a deleted provider
Refreshes run concurrently: after an IAM API change, on a peer's change
and in the startup retry. Each lists the store and then hands STS the
result, so a refresh that listed before a DeleteOIDCProvider could finish
after that call's own refresh and keep the deleted provider trusted until
the next change. Refreshes now hold a lock from the read to the hand-off,
and a startup retry cancelled by installing another store drops its
snapshot instead of applying it.
The retry-cancellation test waits for the retry by polling instead of a
fixed sleep.
* s3/iam: route SetOIDCProviderStore through installOIDCProviderStore
A store installed after Initialize skipped the static-provider overlay
and startup hydration: config-file providers disappeared from the IAM
API, ErrOIDCProviderStatic no longer protected them, and stored
providers were never trusted until the next mutation or peer event.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-29 11:51:39 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: evaluate trust policies deny-by-default
EvaluateTrustPolicy seeded its result with the engine's DefaultEffect,
so a non-matching trust-policy statement set still resolved to Allow
when the IAM config sets policy.defaultEffect=Allow. A caller holding
a validly signed token from a registered provider could then assume a
role its trust policy does not admit.
Trust policies now start from implicit deny, matching AWS semantics and
the pre-d751623 behavior of evaluateTrustPolicy; DefaultEffect still
governs identity-policy evaluation.
Upgrade note: deployments on defaultEffect=Allow whose trust policies do
not match their callers will see those assumptions refused.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: cover trust policy implicit deny under DefaultEffect=Allow
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-29 11:36:01 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The stored entry came through FindEntry, which restores chunk file ids
from their fid form, while an expected entry built from a metadata-log
event still carries the serialized form (file_id moved into fid). The
proto.Equal saw file_id "" against the restored id and refused every
stamp, so remote.sync re-uploaded each entry and the RemoteEntry stamp
never landed.
Clone both sides and run BeforeEntrySerialization before comparing, so
chunks match on their fid and the file_id spelling is ignored; the stored
entry and the request's ExpectedEntry are left untouched.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: synchronize stream pins and release them on transitions
Guard chunkStream.cacher with the ReaderCache lock everywhere: mount
sections share one ChunkReadAt across concurrent reads, and unsynchronized
release could double-unpin. Reads served from the chunk cache now detach
the stream's pin instead of retaining the previous chunk. Eviction prefers
unpinned downloaders so a pinned buffer is not dropped mid-stream. A new
ReleaseStream lets callers drop their pin without destroying the shared
cache; S3 and WebDAV readers use it. lastChunkFid becomes atomic since
concurrent mount reads can update it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep eviction bounded when every downloader is pinned
Both eviction paths still fall back to a pinned victim when no unpinned
one exists, so abandoned stream pins cannot bypass the downloader limit
or stall the memory budget. Budget eviction also rechecks the pin under
the ReaderCache lock at removal time: a stream that pinned the selected
victim in between keeps it mapped and the selection retries.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: restore budget bookkeeping when a victim gets pinned mid-eviction
removeUnpinned losing the pin race left the victim out of the idle list
while still holding its reservation, making it unevictable even as the
pinned fallback. Push it back when the reservation is still live.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The ReaderCache is shared by all streams of a process (every S3 GET, for
instance), but a ChunkReadAt released chunks as if it owned them:
- moving on to the next chunk called UnCache on the previous one,
destroying the buffer even when other streams were still inside it;
- since #11384 a buffer is dropped once any reader has consumed it to the
end and no read call is in flight. Streams copy out in slices (256 KiB
in the S3 gateway), so between two calls a slower stream is not
attached and loses the buffer to a faster one.
Either way the slower stream refetches the whole chunk from the volume
servers. With many clients downloading the same popular object at once,
each chunk is fetched over and over; in production we saw the S3 gateway
pull ~10 Gbit/s from volume servers while serving ~1 Gbit/s to clients.
A ChunkReadAt now pins the chunk it is positioned in. The pin is taken
and released only under the ReaderCache lock, since concurrent ReadAt
calls on one ChunkReadAt (as in mount) share it. It is released when the
stream reads the chunk to its end, moves to another chunk (including one
served from the chunk cache), or falls back to random reads. A buffer is
dropped once no stream pins it and no read is in progress, if it was
consumed or its last stream left it; a read still in flight when the
stream leaves drops it on detach, as UnCache did via destroy. Eviction by
slot limit and memory budget is unchanged.
lastChunkFid is now guarded as well: concurrent ReadAt calls raced on it.
Tests: two ChunkReadAt instances streaming one object in interleaved
slices fetch each chunk exactly once (2-3 times before); leaving a chunk
for a chunk-cache hit or while another read is in flight releases it;
concurrent ReadAt calls on one ChunkReadAt leave no pins behind under
-race.
2026-09-28 21:56:04 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: remove staged EC generation files on teardown and shard delete
The 2PC generation switch stages each run as <base>.ecNN.v<N> plus
versioned .ecx/.ecj/.vif files. Nothing on the volume server removes
them: isEcDataShardFile only recognises the exact .ecNN name, so the
staged files are invisible to every bookkeeping pass, and even
full_teardown's wipe-all path left them behind. Each re-encode therefore
leaks a full shard set per shard-holding disk.
RemoveEcGenerationFiles sweeps <base>.ec*.v<N> and <base>.vif.v<N>,
optionally keeping generations at or above a threshold; teardown and the
reconcile wipe remove every generation, and a per-shard delete removes
that shard's staged generations too.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: delete staged EC generations older than N via VolumeEcShardsDelete
After a 2PC generation switch commits, the superseded generation's
<base>.*.v<N> files sit on disk with no cleanup path: teardown removes
everything, and a per-shard delete only touches the named shards, so the
executor had no RPC that reclaims just the staged leftovers.
delete_generations_older_than removes staged generation files strictly
below the threshold on every disk. Versioned files are never mounted, so
nothing is unloaded first; the committed generation and the canonical
files are preserved.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: mirror staged EC generation cleanup
Parity with the Go volume server: remove_ec_generation_files sweeps
<base>.ec*.v<N> and <base>.vif.v<N> staged by the 2PC switch, called by
remove_ec_volume_files (which covers both teardown paths) and the new
delete_generations_older_than request field; delete_ec_shards removes a
shard's staged generations along with the canonical file.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: match staged generation filenames literally
filepath.Glob interprets metacharacters in the collection part of the
base name, so a collection like a[bc] could match another volume's
staged files (or miss its own). Scan the directory and compare names
literally instead, mirroring the Rust read_dir implementation.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: report generation-sweep errors and drop the store lock first
- snapshot the location base names under the read lock and run the
filesystem sweep after dropping it, so a slow disk cannot stall the
store;
- record per-entry read_dir errors in remove_ec_generation_files and
propagate them from remove_ec_shard_generations instead of flatten()
skipping them;
- warn when a staged-shard generation fails to delete rather than
reporting success with files left behind.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: fail shard delete when the staged-generation listing fails
A transient ReadDir failure fell back to removing canonical shard names
only: staged .v<N> files survived while the RPC still reported success,
leaving the leak invisible to retrying callers. ENOENT still means the
disk simply has no such directory; other listing errors now propagate.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: propagate staged-generation removal failures
delete_ec_shards logged remove_ec_shard_generations errors and the RPC
returned success while staged .v<N> files remained, diverging from the
Go handler which surfaces the failure. The sweep keeps processing the
remaining shards, retains the first error, and volume_ec_shards_delete
maps it to Status::internal so callers can retry.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: notify state change even when the shard sweep errors
delete_ec_shards already deletes and unmounts the shards before
returning a staged-generation failure, so returning early skipped
volume_state_notify and the master kept routing to them until the next
heartbeat. Notify before propagating the error.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: make volume capacity reservation timeout configurable (#11426)
* master: expire reservations on reads, fix int timeout units
- AvailableSpaceForReservation now expires reservations too: a node that
is full of reservations is filtered out before TryReserveCapacity can
clean them, which stranded expired capacity indefinitely.
- Drop TryReserveCapacityWithTimeout: a per-call timeout lets one caller
expire another's live reservations, and the Node interface stays
stable for implementations outside this tree.
- parseReservationTimeout no longer routes integer values through
GetDuration, which read them as nanoseconds; bare numbers are
seconds. The 5m fallback is now the shared DefaultReservationTimeout.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3: do not promote ?prefix into the object for non-List actions
authRequestWithAuthType mapped an empty object to the prefix parameter for
every action, so PUT /bucket?versioning&prefix=x authorized as Write:bucket/x.
An object-scoped grant (Write:bucket/*) could then change bucket versioning,
lifecycle, cors, and object-lock configuration, and the promoted object also
made ResolveS3Action report s3:PutObject to attached IAM policies.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: treat GET ?uploads as a bucket listing for authorization
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: resolve the listing action through the bucket-level object
resolveS3AuthTarget fed the promoted prefix to ResolveS3Action, so a
bucket-level ?uploads request resolved as s3:GetObject on the prefix ARN
in the admin explicit-deny check. Resolve both action and resource
against the object the bucket listing actually scopes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: resolve the listing action through the bucket-level object in AuthorizeAction
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: drop the unreachable object-level uploads case from the resolver test
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-28 07:17:58 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* s3: a list marker that sorts past the prefix leaves nothing to list
AWS scopes a listing to keys under Prefix; StartAfter, Marker and
continuation tokens only reposition inside that range. A marker that
diverges from the prefix at a larger byte is after every key the prefix
can match, so the page is empty. normalizePrefixMarker used to keep such
a marker as the walk cutoff at the bucket root, where the walk descends
into the marker's own directory and returns keys the prefix never names.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep the listing variant's action when a prefix is promoted to object
authRequestWithAuthType promotes ?prefix= into the object argument for
the legacy CanDo path. ResolveS3Action treats a non-empty object as
object-level, so a bucket-level ?versions or ?uploads request carrying a
prefix missed its specific action and fell back to the base List action:
an s3:ListBucket grant then covered s3:ListBucketVersions, and an
explicit Deny on the specific action was skipped on the same path.
Resolve the action against the same bucket-level object the resource
ARN already uses.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: treat GET ?uploads as a bucket listing for authorization
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Update weed/s3api/auth_credentials.go
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* s3: ignore empty intermediate directories in bucketHasUserObjects (#11490)
* s3: keep nested reserved-named dirs from hiding user objects
Reserved folders (.uploads, *.versions) are internal only at the bucket
root; deeper entries with those names are user key prefixes and must be
walked. Also treat a missing subdirectory as empty via isFilerNotFound
(list errors cross gRPC as status errors, not the sentinel), let names
containing backslashes count as objects, and walk iteratively so empty
chains deeper than the old scan depth no longer report non-empty.
* s3: treat reserved-named directories as internal at every level
Object listing interprets .uploads and *.versions directories as
internal storage wherever they appear, so walking them during the
emptiness check would report invisible version remnants as user objects
and block deletion. A reserved name on a file still counts, matching
listing which only special-cases directories.
* s3: count explicit directory objects under reserved names
A directory object created by PutObject (MIME or prefix-object marker
set) is user data even when named .uploads or *.versions; only a plain
directory with a reserved name is internal storage.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-28 06:56:46 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: require dedicated object-lock permissions for x-amz-object-lock-* headers
PutObject, CreateMultipartUpload, and PostPolicy honor the retention and
legal-hold headers after only the route's s3:PutObject check, so a
write-only principal could pin a version under COMPLIANCE retention that
nobody can remove before its retain-until date. On AWS these headers
require s3:PutObjectRetention / s3:PutObjectLegalHold. validateObjectLockHeaders
is the shared funnel for all four call sites; it now authorizes the
corresponding dedicated action when each header is present.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: record the verified POST-policy signer as the request identity
The handler authenticated the form policy signature but stored only the
signer's name, so downstream authorization (the object-lock header check)
re-authenticated the form-signed request as anonymous and evaluated the
wrong principal.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.
The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
volume_copy was one ~400-line handler, and the rule that an existing
local replica is deleted only after the source's ReadVolumeFileStatus
succeeded was held by statement order alone. The keep_remote_data=true
that the pre-copy delete and the failed-copy rollback must share was
kept in sync by a comment pointing from one to the other.
The handler is now a ~60-line orchestrator over connect_to_copy_source,
SourceVolumeStatus::fetch, delete_existing_replica,
plan_copy_destination and a VolumeCopyJob whose run() drives
preallocate_dat, transfer_files, finish_copied_files and
mount_and_reply, with cleanup_failed_copy on error.
delete_existing_replica takes a &SourceVolumeStatus, which only fetch
can construct (private field in a child module), so the delete cannot
be called before the status RPC. Both deletes go through
delete_replica_keep_remote.
Pure refactor: call order, status codes and messages, cancellation
checks, throttling, progress reports and cleanup are unchanged.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix(volume-rust): reserve a disk before replacing a replica in VolumeCopy, and check record counts
Port of the Go VolumeCopy hardening in #11238 and #11252.
- Pick the destination disk before deleting the existing replica, counting
the slot that replica holds as free. If no disk qualifies, the healthy
replica is kept instead of being deleted.
- Read the source's VolumeStatus before and after the copy. When both
succeed and the counts did not change, the mounted replica's file and
deleted counts must match; on mismatch it is unmounted and its files
removed. A failed "before" read skips the check; a failed "after" read
fails the copy.
* fix(volume-rust): let a departing caller cancel VolumeCopy's post-copy status read
Go reads the source's status after the copy with stream.Context(), so the
call ends when the caller leaves. The Rust call had no such link: a
source that stalled there held the copied, unmounted files after the
caller was gone. Race it against the response channel, like the other
blocking steps, so the usual error cleanup removes the partial copy.
remote_storage/s3_tier.rs returned Result<_, String> from every
transfer (upload_file, download_file, read_range[_blocking],
delete_file[_blocking]) and from the tier runtime helpers. The tier
move handlers could only wrap that in Status::internal, so a .dat whose
remote object is gone was indistinguishable from an I/O failure to
weed shell.
Add TierError { NotFound, Io, RuntimeUnavailable, Aborted }. Each
variant carries the existing message verbatim. NotFound follows the
rules remote_storage/s3.rs already uses: raw 404 status on HEAD,
NoSuchKey code on GET; a bare 404 on GET stays Io. A progress-callback
Err becomes Aborted. VolumeError gains a transparent Tier variant and
From<VolumeError> for Status maps Tier(NotFound) to NotFound; the tier
move handlers go through status_with_context, so their message text is
unchanged. Every other tier failure is still Internal.
The remote needle read path keeps io::Error::other, so its error kind
and vacuum's handling of it do not change.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: run the vacuum compaction copy without the store lock
VacuumVolumeCompact held the store write lock for the whole live-needle
copy, including every progress blocking_send on the 16-deep stream. On a
large volume that is minutes with every read, write and heartbeat on the
node parked behind it, long enough for the master to unregister the node.
Split compaction the way Go's CompactByIndex runs it. A short locked step
claims the volume's compacting flag, records the makeup_diff watermark
(index size and compaction revision) and opens fresh .dat/.idx handles.
The copy then replays .idx up to the watermark and copies from those
handles with the store lock released; writes that land meanwhile are
replayed by makeup_diff at commit, as before. The flag is an Arc<AtomicBool>
released when the job is dropped, so every exit path clears it.
Because the flag is now visible to other callers, the operations that would
pull the files out from under the copy refuse while it is set: unmount
(and VolumeConfigure, which unmounts and remounts), delete (checked before
the volume is removed from the map, which a refused destroy used to leave
unmounted), cleanup, and index relocation. A second compact and a commit
stay no-ops, as in Go. The pre-copy fsync is dropped: the copy reads its
own handles through the page cache and .cpd/.cpx are fsynced before commit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: keep a read-only in-memory index's size for the compaction copy
The unlocked copy replays .idx up to index_file_size(). A read-only
volume whose .sdx could not be built loads its index into memory without
a writer, so that size stayed 0: the copy came out empty and the commit
replaced the volume with it. CompactNeedleMap::load_from_idx now records
the rows it loaded, which is also what Go's IndexFileSize reports for a
read-only index.
The copy's index replay now stops reading at the recorded size instead
of walking rows appended since, which makeup_diff replays anyway.
Adds tests for compacting a read-only volume on both the sorted index
and the in-memory fallback, and for VolumeConfigure stopping when the
unmount is refused during a copy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stop a vacuum copy as soon as its client is gone
The progress callback only noticed a closed response stream when a report
was due, every 128 MiB. With the copy now running outside the store lock,
a copy nobody waits for keeps the volume marked compacting and so keeps
refusing unmount, delete and cleanup until that next report. Check the
stream on every callback.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stream ReadAllNeedles without holding the store lock
read_all_needles held store.read() while Volume::read_all_needles read
every live needle of the volume into a Vec, and kept holding it through
the whole blocking_send loop. Memory grew with the volume, and a slow
client parked the scan in a send with the guard held; needle writes and
the heartbeat take store.write() on a writer-preferring lock, so the node
stopped serving until the client caught up.
Take a DatScanPlan (fresh .dat open, end bound) under a short guard and
walk it with the guard released, sending one needle at a time. Each
record is checked against the live needle map under a brief read guard,
as the scan reaches it, and only a live record is parsed, so a damaged
stale copy does not fail the stream. Records appended while a pass ran
are walked by a follow-up plan, so a needle overwritten during the scan
is streamed once, as its new copy. A vacuum commit or re-create of the
volume during the scan fails the stream, since the map's offsets no
longer describe the pinned file; the plan carries the volume instance
and compaction revision for that check.
DatScanPlan::scan_records yields records unparsed; scan keeps its
behaviour on top of it. Volume::read_all_needles has no caller left and
is removed; its tests move to the RPC.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: check ReadAllNeedles liveness only once the send can proceed
The per-record liveness check ran before blocking_send, so a scan parked
on a full channel held a record it had already judged live. An overwrite
landing during that park left the old copy in the stream, and the
continuation over appended records then streamed the new copy as well.
Reserve channel space first, then take the store read guard, check the
record against the needle map and enqueue it through the permit before
releasing the guard. The wait for space still happens without the lock;
the record is parsed before the guard is taken, and its parse error only
counts if the record turns out to be live.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: collect heartbeats under the store read lock, off the runtime
Every pulse tick, options change and volume-state notification took
store.write() for a whole heartbeat pass, directly on the async heartbeat
task. The pass fstats every volume's .dat twice and hashes its report, so
on a server with many volumes it held the store exclusively for the whole
scan: reads and writes stalled, and with the writer-preferring RwLock a
pending pass parked every new reader too.
The pass only needs to mutate the store for a few rare actions: removing
expired EC volumes, deleting expired volumes past their removal delay, and
setting no-write on IO-quarantined volumes. It now runs under store.read(),
records those as (disk, volume id) actions, and applies them afterwards
under a short store.write() that is only taken when there is something to
do. Each action re-checks its target under the write lock, so a volume
written to, replaced or removed in between is left alone. Expired EC
volumes are still removed before the volume pass, as before, because the
EC shard count feeds the disk-space-low max volume count. Every pass runs
on the blocking pool via spawn_blocking.
The heartbeat message is unchanged for the same store state.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: take has_no_ec_shards with the heartbeat's volume list
The heartbeat pass took has_no_ec_shards from the EC phase's read lock,
then built the volume list under a second one. An EC shard mounted in
between went out as "no EC shards" beside a volume list taken after the
mount, and the master clears a server's EC registrations on that flag.
has_no_ec_shards is now computed under the same read lock as the volume
list, with the EC phase's filter: not expired, not quarantined, at least
one shard.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: let the sweep release volumes that stay empty and quiet
Vacuuming reclaims bytes but not slots: a fully emptied volume stays
registered to its collection forever, and since growth is gated only on
slot count a store at 99% free disk can still refuse writes to other
collections (#11429). volume.deleteEmpty exists but is manual-only.
With -vacuumDeleteEmptyAfterSeconds (or master.vacuumDeleteEmptyAfterSeconds
under weed server/mini; default 0, off) the automatic sweep now deletes
replica copies that have stayed empty and quiet for that long, the same
rule volume.deleteEmpty applies on demand: remote-backed copies are
skipped, and every delete carries the volume server's onlyEmpty /
onlyGarbage guards so a copy written since the last report is refused
rather than removed. Copies that still hold data or were written
recently stay; only a volume whose every copy is deleted leaves the
sweep's work map, sparing a compaction of bytes that are all deleted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: harden empty-volume sweep against partial and racing deletes
Review follow-up on #11477:
- delete a volume only when every replica copy is a verifiable
empty-and-quiet candidate; deleting the empty copy of a volume whose
sibling holds live files would silently cut its replica count
(greptile P1).
- drain the volume out of the writable list before deleting, the same
drain the compact pass uses, so PickForWrite stops assigning it and
pending writes settle (devin).
- bound the VolumeDelete RPC so one stalled server cannot hold the
vacuum lock indefinitely (greptile P1, reusing allocateVolumeTimeout).
The vid2location panic scenario raised in review does not exist:
VolumeLocationList methods are nil-receiver safe and a missing vid just
fails enoughCopies, so a partially deleted volume skips compaction
instead of crashing the sweep.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: unregister deleted empty replicas and prune the sweep list
A successful VolumeDelete only updates the volume server; the master
still tracked the replica and kept it in the sweep's location list for
the compaction pass (coderabbit on #11477). Unregister the replica right
after its delete succeeds and drop it from the sweep copy, so a partially
deleted volume only compacts copies that still exist.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: pin deleting volumes out of the writable list across heartbeats
Review follow-up on #11477 (greptile): DrainAndRemoveFromWritable only
removed the volume once; a heartbeat landing between the drain and the
replica deletes re-evaluated writability and re-added it, so a client
write could reach a replica whose siblings were already gone and leave
the volume under-replicated when the last copy refused its onlyEmpty
delete.
MarkDeleting records the vid in deletingVolumes — checked inside
setVolumeWritable so heartbeat, capacity-recovery, and admin re-add
paths all hold it out — and UnmarkDeleting releases it once the sweep
finishes the copy pass. A partially deleted volume's surviving replicas
then return to writable through the normal heartbeat path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: restore writability when a sweep delete survives
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-27 07:05:22 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: record requester identity before the authz verdict for audit
Identity was only stored in request context on the success branch, so
denied requests reached WriteErrorResponse without requester attribution
and audit entries had empty requester/requester_arn/requester_identity.
Authentication failures still resolve no identity, so unauthenticated
denials stay unattributed.
Fixes#11474
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep the resolved identity through authz denial in Auth
Review follow-up on #11479 (devin): authRequest discarded the identity
on every error, so a request that authenticated fine but failed the
action check still reached handleAuthResult with no identity and the
deny path could not audit a requester. Auth now calls
authRequestWithAuthType directly, the same entry AuthPostPolicy uses,
so the resolved identity reaches the error writer; a failed authN
still resolves no identity and stays unattributed. The regression test
now signs a denied request end to end through iam.Auth.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-27 07:03:01 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sts: export CalculateSessionDuration
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: honor configured session bounds on AssumeRole and LDAP identity
prepareSTSCredentials hardcoded a one-hour session when the caller
omitted DurationSeconds, so sts.tokenDuration was ignored and
sts.maxSessionLength only clamped explicit requests: asking for 3600s
against a 20m ceiling was rejected while omitting the parameter was
granted a full hour (#11473). The two affected handlers now use the
same default-then-cap calculation as AssumeRoleWithWebIdentity.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: keep MaxSessionDuration through role store copies
copyRoleDefinition rebuilt RoleDefinition field by field and dropped
MaxSessionDuration, so memory-backed role stores silently discarded the
per-role session bound on every write and read (devin on #11478).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sts: apply per-role MaxSessionDuration to resolved session durations
Review follow-up on #11478 (devin): the role bound only ever applied to
explicit DurationSeconds values — an omitted duration resolved to the
configured default and sailed past a shorter role max on every assume
path.
- capDurationByRole now resolves min(requested||tokenDuration, roleMax),
so AssumeRoleWithWebIdentity and AssumeRoleWithCredentials cap
defaults the same way they cap explicit values.
- prepareSTSCredentials caps the calculated duration at the named
role's MaxSessionDuration, covering the AssumeRole and LDAP handlers;
self-assumption has no role definition to consult.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: keep MaxSessionDuration through the cached role store
genericCopyRoleDefinition drops MaxSessionDuration the same way
copyRoleDefinition did, so the cached filer role store reads back a zero
maximum and every downstream duration cap is skipped (greptile on
#11478).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sts: only materialize defaults that pass session duration validation
Review follow-up on #11478 (greptile): materializing an omitted
DurationSeconds into an explicit value could exceed the service's own
input bound (a configured tokenDuration above maxSessionLength) and turn
a previously working request into a validation error.
capDurationByRole now leaves nil anything the service can resolve
better itself, clamps a tightened default at maxSessionLengthSeconds,
and floors a role bound below 900s to the tightest issuable value.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-27 07:01:51 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep dedicated object-lock actions pinned during action resolution
A coarse action that already names a dedicated operation (governance
bypass, retention, legal hold, bucket object-lock config) now resolves to
itself before request shape is consulted. Previously a synthetic
DELETE ?versionId authorization request re-resolved to
s3:DeleteObjectVersion, so the bypass check was satisfied by the
delete-version grant alone; with the pin it evaluates
s3:BypassGovernanceRetention as intended.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: cover pinned object-lock actions against competing query params
Locks in the resolution for every dedicated action in the pin set, incl.
the retention and legal-hold shapes carrying versionId.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: stamp entries with IF_CHUNKS_EQUAL so a stale write-back cannot delete live chunks
updateLocalEntry records the RemoteEntry stamp after an upload by writing the
event's entry back with UpdateEntry. The filer deletes every stored chunk
absent from an updated entry, so when the file was rewritten while its upload
was in flight (or the event is a replay), the stale snapshot deletes the
rewrite's chunks: the entry then points at the new fid with no needle behind
it, and the rewrite's own upload fails and is skipped as superseded.
The stamp write now carries WriteCondition IF_CHUNKS_EQUAL over the event's
chunk fids, evaluated by the filer under the path lock. A refused stamp means
the filer moved past this event; the superseding event follows in the log and
stamps the current entry, so the refusal is logged and skipped like a
superseded upload.
Reproduction: weed server -filer plus a weed server -s3 remote, remote.mount,
filer.remote.sync; hold the remote (docker pause) so one upload stays in
flight, rewrite the file through the filer, unpause. Before: the entry's chunk
is 404 on every volume server. After: the stale stamp is refused, the rewrite's
chunk stays live and reads back after a vacuum.
* filer.remote.sync: stamp entries with IF_ENTRY_EQUAL so stale inline content or metadata cannot be restored
The IF_CHUNKS_EQUAL guard compared only the chunk fid multiset, so a
rewrite that touched inline content or metadata alone still compared
equal and the stale snapshot overwrote the live entry. The new clause
compares the whole stored entry against the event's entry under the
same path lock.
* filer: route conditional UpdateEntry to the entry's owner filer
Two filers locking the same path locally could still pass a stale
condition on the non-owner while the owner's entry had moved on. When a
condition or expected_extended precondition is set, forward the request
to the entry's owner the same way conditional CreateEntry does, with
is_moved bounding the hop.
* filer: compare IF_ENTRY_EQUAL against the normalized expected entry
FindEntry grows FileSize to the chunk extent, so a raw event entry with
FileSize still zero failed the condition on an unchanged file and the
stamp was skipped, letting a replay upload the object again.
* filer.remote.sync: classify refused stamps by gRPC status only
A FailedPrecondition substring in an unrelated error would have been
swallowed as a skipped stamp; status.FromError already unwraps.
* remote sync: keep the event entry intact for IF_ENTRY_EQUAL
* filer: honor is_moved only from ring member connections
is_moved is caller-controlled, so a request could set it to skip owner
routing and run a conditional check under a non-owner's lock. Verify the
marker against the peer's connection address and the lock ring members;
an unverified marker is ignored and the request routes like a fresh one.
* filer: refuse unverifiable is_moved at a non-owner, cache ring IPs
Follow-up fixes from review on the is_moved provenance check:
- checkMovedMarker replaces "ignore and re-forward" for markers that did
not arrive on a ring member's connection. Re-forwarding a claimed hop
could cycle while rings disagree; instead the request is refused with
FailedPrecondition unless this filer is the key's owner, in which case
applying locally is correct anyway.
- ringMemberIPs caches resolved member addresses per ring membership so
hostname-advertising deployments do not pay a DNS lookup per forwarded
request; failed lookups are not cached so a DNS blip self-heals.
- DistributedUnlock no longer dereferences the nil response of a failed
next-hop RPC.
* filer: refuse unverifiable is_moved with PermissionDenied, not FailedPrecondition
A routing refusal is different in kind from a write-condition mismatch:
remote sync treats FailedPrecondition as a stale stamp and skips it, so
reusing that code let a routing failure pass as synced. Owner checks now
also run before the peer-IP lookup so the common accept path does no DNS.
* filer: expire resolved ring member IPs after 5 minutes
A member's hostname can re-resolve to a new IP while its ring address
stays unchanged; caching forever would reject its genuine forwards until
a membership change or restart.
* filer: deduplicate concurrent ring member DNS lookups
At cache expiry, parallel forwarded requests would each resolve every
member hostname serially; singleflight collapses them into one lookup
per ring membership.
* filer: detach the shared ring lookup from the caller's context
The singleflight winner's ctx is cancelled when its request ends; the
shared result would then be an incomplete member list and genuine
forwards denied. The lookup now runs on a detached context with its
own deadline so a canceled caller cannot poison it.
* filer: resolve ring member hostnames in parallel
The shared lookup gave every member one serial budget, so a few slow
resolutions could leave later members out of the cached list and reject
their genuine forwards. Each member now resolves concurrently under its
own detached deadline.
* filer: gather literal member IPs before spawning lookups
A ring mixing IP literals and hostnames raced: the literal appends ran
unlocked alongside the resolver goroutines' locked appends. Split into
two passes so only hostname results share the mutex.
---------
Co-authored-by: jsas <1351492+jsas@users.noreply.github.com>
2026-09-26 19:42:23 +08:00
Mohd Quamar TyagiGitHubcoderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>Chris Lucoderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* volume: skip TTL last-write scan when it cannot fit its budget
* Update weed/storage/volume_checking.go
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* s3: evaluate bucket policy before ACL public-read for anonymous requests
AuthWithPublicRead granted anonymous access on a public-read ACL before
consulting the bucket policy, so an explicit Deny (e.g. s3:ListBucket)
was skipped for anonymous callers while still enforced for authenticated
ones. Run the policy engine first: a matching Deny or Allow is honored,
otherwise fall through to the ACL grant as before.
* s3: defer object-level anonymous requests to the handler's policy recheck
Evaluating the bucket policy with a nil entry at middleware time makes
tag conditions like s3:ExistingObjectTag/<key> resolve against missing
values, so a conditional Deny could wrongly block anonymous Get/Head on
a public bucket whose handler recheck would permit it. Object requests
now take the ACL grant and let Get/HeadObjectHandler re-evaluate with
the fetched entry; only bucket-level requests (List, HeadBucket), which
have no such recheck, are decided by the middleware policy verdict.
Reading the bucket config first also refreshes the compiled policy on a
cache miss, so a remotely deleted policy cannot leave a stale verdict
in the engine for nonresident buckets.
* s3: recheck bucket policy before serving directory objects
handleDirectoryObjectRequest runs before the object handlers' policy
recheck, so directory content on a public-read bucket was served to
anonymous callers without any policy evaluation. Evaluate the policy
with the directory entry, matching the recheck the file path performs.
2026-09-26 17:59:57 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: hold back last chunk until CRC verifies on whole-needle reads
Above PagedReadLimit the needle is streamed: headers and body go out
before the checksum is computed, so a corrupted needle was served as
200 with bad bytes and readers could not fall back to a replica. The
final chunk is now written only after the checksum verifies; on a
mismatch the response ends short of Content-Length and the client sees
a failed transfer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: verify needle checksum on streamed reads (parity)
Mirror the Go fix: carry the needle checksum in NeedleStreamInfo and
have StreamingBody accumulate the CRC and verify it before emitting the
last frame; a mismatch ends the body with an error so the client sees
the transfer fail rather than receiving corrupt bytes that look
complete.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* server: abort the transfer when a committed stream fails
A writeFn failure after any byte or WriteHeader call leaves the declared
status and Content-Length already sent; http.Error's text then joins the
body and can exactly fill the withheld tail of a corrupted needle read —
the client sees a complete 200 instead of a failed transfer to retry.
Track whether the response is committed (headers sent, or bytes buffered
for the deferred flush) and panic with http.ErrAbortHandler instead of
appending an error body; pre-commit failures keep the 500 path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* server: drop the response writer wrapper from the committed-response check
Counting buffered writes is enough: with no bytes buffered the status and
headers cannot have gone out, and the range branches commit via the
explicit WriteHeader call before writeFn runs. The extra ResponseWriter
wrapper added a new Write sink site that CodeQL flags.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 16:50:41 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: scrub live needles' stored id against the index key
scrubVolumeData only compared the needle's stored id for tombstones, so
header damage on a live needle — where the data CRC cannot see it —
passed every scrub mode while reads of that needle kept failing or
serving the wrong key's data. Compare the id for every indexed needle.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: scrub live needles' stored id against the index key (parity)
Mirror the Go scrub fix: compare the stored needle id with the index
key for live needles too, not only for deleted ones.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: cover damaged live needle id in scrub test
The tombstone test proved the index-key check fires for deleted entries;
add the live-needle mirror of Go's TestScrubVolumeDataChecksLiveNeedleId
so a regression in the live path is caught in Rust too.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 16:10:14 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: serialize every ViperProxy method; stop promoting unlocked viper calls
ViperProxy embedded *viper.Viper, so only the five declared methods took
the mutex while every promoted call — GetStringMap in backend.LoadConfiguration
was the reported crash — touched viper's maps unsynchronized. `weed server`
starts the volume server (SetDefault writer) and the master (GetStringMap
reader) back to back, and a race build reports the pair on a plain start.
The wrapped viper is now a named field: a method must be declared here to
exist on the proxy, so unsynchronized access fails at compile time rather
than at runtime. Every promoted use in the tree (GetStringMap, GetUint32,
GetFloat64, GetDuration, IsSet, AllKeys, Set) gets a locked wrapper;
NewViperProxy replaces struct literals for local vipers. GetStringMap
deep-copies its result — viper hands back the internal subtree, so
iterating it after the lock is released would race the next writer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: take the shared lock while LoadConfiguration merges a config file
viper.MergeInConfig rewrites the same maps the proxy serializes; without
the lock a merge can race a concurrent SetDefault or reader exactly like
the reported startup crash.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: deep-copy slice elements in the GetStringMap snapshot
A slice of maps inside the returned subtree still shared the inner maps —
copy elements recursively so nothing the caller mutates is viper's
internal state.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: add the missing AutomaticEnv wrapper used by tests
sse_reader_test reaches it through GetViper(); without the wrapper the
call no longer exists once the viper field stopped being embedded.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: return a fresh slice from GetStringSlice
A stored []string comes back uncast from viper — the backing array is
shared internal state like the GetStringMap subtree, so copy it while
holding the lock.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: do not finish a GET when the needle CRC mismatches
A streamed full-needle read compared the CRC only after every page had been written. Once the response buffer flushed, the client already had a completed 200 and the corrupt bytes. Hold the last page until the checksum matches, and if an earlier page has already been flushed, abort the connection instead of calling http.Error.
Fixes#11459
* volume server: abort partial-content bodies on write error too
The non-Range path drops the unflushed tail and aborts on a mid-body
error; the single-range and multi-range paths still flushed it after
WriteHeader(206) was committed, delivering corrupt bytes as a complete
body.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: assert the started 200 is aborted in the write-error test
The test previously returned on any request error, so it passed without
verifying the abort. It now asserts the client got the committed 200
headers and then a failed body read. Also trims comments.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 16:00:22 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: let volume.deleteEmpty drop volumes with no live needles
The candidate check only accepted a .dat at superblock size, so a volume
whose every needle was deleted still had to be vacuumed first — minutes
of compaction to rewrite bytes that were all garbage anyway. FileCount
counts every indexed entry and DeleteCount every entry made garbage by
overwrite or delete, so FileCount <= DeleteCount means nothing live
remains and the volume can be unlinked directly. The quietFor guard is
unchanged.
* volume server: add only_garbage VolumeDelete guard
VolumeDelete(only_empty) refuses every volume that ever held data, so a
volume whose needles are all deleted could only be removed after a
vacuum rewrote it. The new only_garbage flag deletes only when the byte
counters show nothing live: DeletedSize covering all of ContentSize, the
same all-garbage state vacuum measures. Byte counters are used because
the file/delete counts drift on index reload.
* rust volume: mirror only_garbage VolumeDelete guard
Same check as the Go server: a volume deletes under only_garbage when
its deleted bytes cover all content bytes. The grpc handler rejects
before the store drops the volume from its map, since destroy errors
after removal would still unmount it.
* volume delete: let either enabled check pass, keep onlyEmpty on the wire
An upgraded shell sending only_garbage to a pre-upgrade server would be
read as an unconditional delete (field ignored, only_empty false). The
request now keeps only_empty set so old servers check emptiness and
refuse, while new servers delete when either check passes.
* volume.deleteEmpty: skip remote-backed and protected read-only volumes
A remote-tiered replica shares its cloud object with the other replicas,
so keepRemoteData=false on one delete removes data they still reference.
Protected read-only volumes are quarantined or under maintenance, which
is exactly when a replica should not be dropped.
* volume delete: validate guarded copies across disks before deleting
* volume delete: hold copy locks across guarded validate-and-delete
CheckVolumeDeletable released each copy's locks before Destroy ran, so a
write landing on a later copy between the two passes refused its destroy
after earlier copies were already removed. Pin every copy's
dataFileAccessLock (and its location's volumesLock) across validation and
removal so a refused delete leaves all copies intact.
* volume delete: send deleted-volume notices after releasing locks
A blocking send on a full DeletedVolumesChan under volumesLock can stall
the heartbeat loop that drains it while it waits on the same locks.
Collect the notices under the lock span and send after release.
* pb: restore generated-file cosmetics to match the repo's protoc version
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: one long-lived tokio runtime for blocking tiered S3 reads
block_on_tier_future, behind read_range_blocking and delete_file_blocking,
spawned an OS thread and built a fresh current-thread tokio runtime on every
call, then tore the runtime down. On an S3-tiered volume that is once per
needle read, per streamed 64 KiB chunk, per DatScanPlan record and per
destroy. The SDK client's pooled HTTPS connections are driven by tasks on the
runtime a request ran on, so each teardown dropped the pool and every call
re-dialed and re-handshook TLS. A panic inside the SDK was also flattened to
the fixed string "tier runtime thread panicked".
Now one process-wide runtime (OnceLock, multi_thread, 2 workers named
tier-io) drives all tier I/O; block_on_tier_future spawns onto it and parks
the caller on an mpsc channel for the JoinHandle result. Blocking the caller
is unavoidable (the storage layer is synchronous) and is what the old code
did through thread::spawn().join(). Handle::block_on is not used because the
wrappers are also reached from inside another runtime's worker, where it
panics with "Cannot start a runtime from within a runtime". JoinError panics
are downcast to &str/String and the payload is kept in the error.
Tests cover runtime reuse (Handle::id equal across calls, thread name
tier-io), calls from a std thread, from spawn_blocking, and directly from
current-thread and multi-thread runtime contexts, and the panic payload.
Against the old body 7 of 9 fail.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: return a tier runtime build failure instead of panicking
Review follow-up. tier_runtime() expect'ed the runtime build, so an OS
refusing threads panicked inside Volume::destroy (after the volume left
the in-memory map, before its files were removed) and inside needle
reads, bypassing their error paths. Keep the runtime in a
Mutex<Option<Runtime>> behind tier_handle() -> Result<Handle, String>:
a failed build is returned to the caller through block_on_tier_future's
existing Result and is not cached, so a later call retries once the
pressure is gone. The lock is held only while building.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: trim comments on the shared tier I/O runtime
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: VolumeEcShardsDelete full teardown unloads every disk and keeps the shard gauge honest
Go's VolumeEcShardsDelete full teardown calls vs.store.UnloadEcVolume in
the blanket path (weed/server/volume_grpc_erasure_coding.go:488) and
location.UnloadEcVolume in the generation-fenced path (:511): each disk
that had the volume registered drops it, closes its shard descriptors
and gives back its ec_shards gauge before the artifacts are unlinked.
The Rust handler used Store::remove_ec_volume / DiskLocation::remove_ec_volume
instead, which only remove the map entry. Store::remove_ec_volume also
stops at the FIRST disk holding the vid, so on a split-disk volume
(shards on several disks) the blanket teardown left the sibling disks'
EcVolume registered with open fds while the unlink loop deleted their
files underneath it: the heartbeat kept advertising shards whose files
were gone, the inodes stayed pinned by the open descriptors, and the
VOLUME_GAUGE{collection,"ec_shards"} never came back down. The fenced
path leaked the gauge and the descriptors the same way on the one disk
it wiped.
Both paths now use the unload_ec_volume helpers from #11413 (every disk
for the blanket teardown, the strictly-older disk for the fenced one),
and the two Status::internal messages name the disk directory like Go's
"... on %s: %w".
Regression tests build a two-disk store with the same vid mounted on
each disk (the SplitDiskEcFixture, which gains a collection knob so the
gauge read is isolated from parallel tests mounting under "") and assert
that a blanket teardown leaves no EcVolume registered on any disk and
returns the gauge to its pre-mount value, and that a fenced teardown
decrements the gauge for the older disk's shard while preserving the
newer disk. Both fail against the previous handler.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume server: trim comments on the EC full-teardown unload path
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: reject invalid -minFreeSpace and duration flags at startup like Go
Symptom: the Rust volume server swallowed invalid values for three
startup flags and ran with a limit the operator did not ask for.
-minFreeSpace=150 became 150 bytes (Go: fatal, percent must be 0..=100),
-minFreeSpace=10GiBx and -minFreeSpace=50B fell through to the 1%
default (Go: fatal), and -inflightUploadDataTimeout=30sec, =abc or =30
silently became the 60 s default (Go: flag.Duration usage error, exit 2).
Go's byte-size units (1,024MB, 42 mib, 100Ki, 1eb) were also not
understood and mapped to 1%.
Go reference: weed/util/minfreespace.go:73-106 (MustParseMinFreeSpace /
ParseMinFreeSpace), weed/util/bytes.go:212-243 (ParseBytes and
bytesSizeTable), weed/command/volume.go:226-227 (flag.Duration, i.e.
time.ParseDuration grammar).
Fix: the two inflight timeout Cli fields are now std::time::Duration
with a clap value_parser that ports time.ParseDuration (units ns/us/µs/
μs/ms/s/m/h, fractions, bare 0, int64-nanosecond overflow), so clap
prints a usage error naming the value and exits 2 at the same layer as
Go's flag package. parse_min_free_spaces returns Result and is built
from Go-parity parse_min_free_space / parse_bytes; resolve_config_with_env
aborts with Go's "The value specified in -minFreeSpace not a valid
value" message, the fail-fast pattern it already uses for --max, --index
and --readMode. No new dependency. Negative durations and NaN percent
are rejected explicitly (Go quirks with no sane Rust representation).
Tests cover the Go fixtures from minfreespace_test.go and bytes_test.go,
the duration grammar, and flag-level rejection through Cli::try_parse_from
and resolve_config_with_env; the flag-level tests were confirmed to fail
against the previous parser.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: say that -minFreeSpace commas separate directories, never thousands
Review follow-up. parse_bytes keeps Go's ParseBytes thousands-comma
support, but the flag is split on commas into per-directory entries
before any entry is parsed (Go's MustParseMinFreeSpace does the same),
so -minFreeSpace=1,024MB is the two entries 1 and 024MB. The doc
comment and the single-entry tests advertised the comma form as if it
were reachable from the flag. Document the split and add a flag-level
test: two entries from parse_min_free_spaces, and resolve_config
aborting on the directory-count mismatch with one -dir.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: trim comments on the strict flag parsing
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 12:00:06 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: restore the HTTP REST API reference
The REST API documentation was lost in the README refactor, and the wiki
only covers the master server API. Add a complete reference for the three
HTTP surfaces — filer paths, master file-id/topology endpoints, and
volume-server content endpoints — generated from the actual handlers and
query parameters.
* docs: correct REST API details flagged in review
Tagging uses Seaweed- headers not query params, the filer recursive
delete option changes the DELETE default, omitted resize mode does not
mean fit, default file mode is 0660, the master redirect is 308, the
listing flag is -dirListLimit, TUS is enabled by default at /.tus, and
-port.public opens the separate read-only listener.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: match the tagging delete example to canonical header names
Go canonicalizes Seaweed-k1 to Seaweed-K1 on write, and the delete list
is compared case-sensitively, so ?tagging=k1,k2 would not match.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: detect S3 not-found by typed SDK errors, not the "service error" string
remote_storage/s3.rs decided ObjectNotFound by matching the Display
output of an aws_sdk_s3 SdkError against "NoSuchKey" / "404" /
"NotFound". In the locked SDK (aws-smithy-runtime-api 1.11.6,
src/client/result.rs:487-497) that Display is a fixed string per
variant, "service error" for every S3 error, so ObjectNotFound was
unreachable: every missing remote object surfaced as
Other("s3 get object: service error") with the real cause discarded.
Go (weed/remote_storage/s3/s3_storage_client.go) uses typed checks:
HEAD (373-374): awserr.RequestFailure with StatusCode() == 404;
GET (436-437): awserr.Error with Code() == s3.ErrCodeNoSuchKey.
read_file now matches SdkError::ServiceError whose GetObjectError
is_no_such_key(); a bare 404 on GET stays a generic error, as in Go.
stat_file matches HeadObjectError::is_not_found() or a raw HTTP 404
status, Go's actual condition. Non-service errors fall through to
Other unchanged. Every SdkError message in s3.rs and s3_tier.rs is
formatted with DisplayErrorContext so the S3 error code and message
survive instead of "service error".
Six network-free unit tests drive the client through a canned
HttpClient (404 NoSuchKey, bare 404 on GET and HEAD, 404 with a
foreign body on HEAD, 403 AccessDenied on GET and HEAD). They need
aws-smithy-runtime-api as a dev-dependency; it is already in the lock
at a single version, so no new crates.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: HEAD not-found is the raw 404 status alone, as in Go
Review follow-up. The HEAD arm also accepted the SDK's NotFound error
code on any status, so a 400 carrying <Code>NotFound</Code> became a
missing object. Go's stat looks only at RequestFailure.StatusCode() ==
404 (weed/remote_storage/s3/s3_storage_client.go:373); do the same. The
raw status still covers the body-less 404 the SDK turns into NotFound
and a 404 whose body names a foreign code. Regression test for the
non-404 NotFound body, which failed against the previous arm.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: trim comments on the typed S3 not-found checks
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: franchb <ru.balloo@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: exclude marker part in listObjectParts pagination
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* s3api: guard listObjectParts marker boundary and enhance pagination test
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* s3api: fold in review feedback from the parallel #11462 fix
Same core fix; this adds the explanatory comment, tightens the overflow
guard to math.MaxInt64, makes the fake filer sort entries like a real
listing, and adds the marker-exclusivity assertions alongside the
pagination walk.
Co-authored-by: yi111 <yi111@users.noreply.github.com>
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: yi111 <yi111@users.noreply.github.com>
2026-09-26 11:58:23 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* cluster: never broadcast an empty lock ring
An empty member list is never a usable ring state, but a delayed
RemoveServer on a former leader can fire after the new leader already
broadcast the recovered ring. That late broadcast carries a newer
wall-clock version, so clients accept the empty ring and permanently
reject the good one.
Skip the broadcast entirely when the member list is empty, keeping the
last non-empty snapshot for reconnecting clients.
* cluster: periodically rebroadcast the lock ring
Ring updates are purely event-driven, so one lost or poisoned update is
permanent until the next membership change — with a single filer that may
never come. Re-arm a per-group timer after every broadcast so the current
leader keeps re-sending the ring; clients reject nothing newer than their
last accepted version, so a re-sent snapshot always heals a stale view.
* filer,s3api: reset the lock ring on master change
Ring versions are per-master monotonic — each master stamps wall-clock
nanoseconds — so a late high-version update accepted from a former leader
makes the new leader's snapshot look stale forever. Detect a leader
change across the reconnect gap (currentMaster is cleared between
attempts, so remember the last served master) and reset the ring to
bootstrap state so the new leader's view always applies.
* cluster: fail lock acquisition when no lock server exists
retryUntilLocked loops forever, so a filer reporting an empty lock ring
wedges every append write indefinitely. Bound only the "no lock server
found" case — ordinary contention is still waited out since the holder
releases eventually. The constructors now return nil on failure: the
filer append path and S3 object writes fail fast, while mounts degrade
to their existing lockless mode.
* cluster: reset only the ring version on master change
Ring versions are per-master monotonic, so a version gate reset is all a
leader change needs. Clearing the whole ring made every filer its own
write owner until the next update and dropped the prior-owner window for
keys the new leader remaps; the last ring now keeps routing until the
new leader's snapshot transitions off it.
* cluster: skip redundant ring installs and defer rebroadcasts
An unchanged member list now only bumps the accepted version instead of
installing a snapshot: periodic rebroadcasts no longer fire the
topology-change callback or restart the prior-owner window. And a
rebroadcast that lands inside a membership stabilization window yields
to the pending timer rather than publishing an intermediate ring.
* cluster,mount: bound lock unavailability, fail ops that cannot lock
Only 'lock already owned' contention retries without bound now; every
other failure — no lock server, or a dead ring member refusing
connections — shares the same unavailability budget, so a ring naming
departed filers can no longer hang a lock forever. Mount open-write,
create, and rename fail with EAGAIN when the required lock cannot be
acquired instead of proceeding without cross-mount serialization.
* cluster: check pending stabilization inside the broadcast critical section
rebroadcast released the mutex between the pending-timer check and
nextBroadcastUpdate, so a membership change arriving in the gap could arm
a stabilization timer while the rebroadcast emitted an intermediate ring.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: acquire path locks before mutating create/rename state
Create took the DLM lock only after the filer create, so a lock failure
returned EAGAIN with an eagerly persisted file left behind. Rename marked
source handles renamed before acquiring locks, so a failed acquisition
left them suppressing old-path flushes for a rename that never happened.
Both now take the locks first; the create's lock is released again if the
entry race loses to another creator and AcquireHandle takes over.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: keep the old-path lock when rename lock migration fails
The migration stopped the handle's lock before acquiring the replacement,
so a nil result left the handle writing with no lock at all. Acquiring the
new-path lock first means failure keeps the existing lock instead of
reporting success with serialization dropped.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: skip new-path rename lock when a handle already holds it
A target file open for write on this mount already carries a lock on
newPath; the lock manager does not grant a second lock to the same
owner, so the rename would wait on itself until the handle closed.
Also avoid locking twice when old and new paths coincide.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: hand the rename's target lock to the migrating handle
The rename holds a lock on newPath for its duration, so the response
migration's fresh acquisition waited on that same lock until the handle
released — under fhLockTable, blocking the handle's own close. Adopt the
rename's lock directly; nested move responses still acquire their own.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: move the replaced target's lock to the renamed handle
When the target path was already locked by an open handle on this
mount, the migrated source handle kept only its stale old-path lock —
the target's close would then release the last lock on the new path
while the renamed handle was still open. Adopt the replaced handle's
lock instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: stop the handle lock inside the fh lock on release
ReleaseHandle stopped fh.dlmLock before taking the fhLockTable slot, so
a rename migration holding that slot could still observe and adopt a
lock that was already stopping. Stopping under the fh lock makes the
transfer serialize against the release.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: claim the replaced target's lock for the renamed handle
When the target path is already locked by an open handle on this mount,
adopting it at migration time keeps the renamed path protected after
that handle closes, without waiting on a lock this mount already holds.
If the handle was released mid-migration the claimed lock is stopped,
and a fresh acquire covers the case where it was already gone.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: claim the target handle's lock before the rename runs
Skipping the new-path lock when a handle already holds it let that
handle's close release the lock mid-rename, leaving the path unguarded
until the response migrated it. Take over the lock at check time and
hold it for the rename's duration: the response adopts it for the
migrating handle, or it returns to the target handle / is released on
failure. The target handle lookup also falls back to the entry's stored
inode for a forgotten path mapping.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: read handle locks only under the fh lock during rename
The loose dlmLock reads raced ReleaseHandle, which now mutates the lock
inside the handle lock; check and claim it under the same hold.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep lazy remote reads from resurrecting deleted paths
Under a remote mount with filer.remote.sync as write-back, a path that
was deleted or renamed away could come back as a chunkless remote-only
entry: between the local delete and the daemon's remote delete, a store
miss made maybeLazyFetchFromRemote trust a bucket that was behind the
filer. The ghost then outlived the remote object -- HEAD answered 200,
GET failed, and nothing cleaned it up.
The filer now tombstones paths it deletes under a remote mount, learned
both synchronously from its own delete path and from peer metadata
events. The lazy fetch and the lazy listing skip a tombstoned path until
the path is written again, until the mount's persisted write-back sync
offset has passed the delete event (the remote delete has landed), or
until a generous TTL covers a mount without a daemon.
Fixes#11440
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: cover recursive remote deletes with an ancestor tombstone
A recursive delete now records the directory tombstone before walking
children, so a partial traversal or a store that drops the subtree
without listing it still leaves every descendant covered. Directory
tombstones also subsume older descendant entries on add, descendant
adds covered by a standing ancestor are skipped, and an existing
tombstone can be refreshed even at capacity.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: scope remote tombstones to the deleted object's generation
A remote object whose own mtime postdates the local delete is a new
generation, not the one the tombstone hides, so a recreated directory
can surface remote writes made after its delete while old-generation
objects stay hidden. Lazy fetch now stats the remote object before
deciding, listings pass each child's remote mtime, and a sync offset
releases a tombstone once it reaches the delete's own timestamp.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: rebuild remote deletion tombstones after restart
In-memory tombstones are lost on restart while remote write-back
offsets persist, so a filer boot replays the persisted metadata log
from the oldest mount offset and folds deletes back into the tombstone
set through the same event handler. Lazy remote reads hold off while
the replay runs so a pending delete cannot resurrect in the gap.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: release remote tombstones only after their delete event lands
The write-back offset orders against event timestamps, but the synchronous
delete path recorded tombstones with the local clock before its event was
emitted — a later unrelated event could already have pushed the mount's
watermark past that guess, releasing the tombstone before the daemon
applied the delete. Tombstones recorded ahead of their event are now
marked pending and can only be lifted by the event confirming them or by
TTL; event-stamped tombstones release through the offset as before.
The remote-mtime generation bypass is dropped: remote and filer clocks
are independent, and a pending remote delete removes whatever object sits
at the path, so a "newer" remote object would only resurrect as a
phantom. Tombstoned lookups now skip the remote stat entirely.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: drop dir tombstone when recursive delete fails before listing
The ancestor tombstone is recorded before the child listing; if that
listing fails nothing was deleted, and the leftover tombstone would hide
still-existing remote children for the whole TTL. Tombstones for children
already deleted stay, since their remote deletes are still owed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: block lazy remote reads on startup tombstone rebuild
The rebuild gate is now a done-channel set synchronously before the
replay goroutine starts, so no lazy read can slip through in between.
Reads wait on it with context cancellation instead of returning an
empty miss that makes remote-only objects look deleted.
The replay start is floored at now-TTL: mounts without a recorded
write-back offset previously replayed the whole persisted history, and
events older than the TTL would only build already-expired tombstones.
The gate check now runs after the mount lookup so replaying the meta
log's own directory listings does not deadlock on the gate, and the
replay retries with backoff until it succeeds instead of failing open.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: mark restamped tombstone pending until its delete event lands
When a local delete raises an existing tombstone's timestamp, the new
value is only a local clock guess ahead of that delete's event. Leaving
the tombstone un-pending lets a write-back offset release it before the
event is actually consumed, reopening the resurrection window.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound tombstone replay to the tombstone TTL
Persisted-log replay retried forever, keeping lazy remote reads gated
indefinitely when the log cannot be read. Cap retries at the tombstone
TTL measured from replay start: past that point every tombstone would
have expired anyway, so opening the gate loses no protection.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: re-check deletion tombstone before persisting lazy fetch
A delete landing while StatFile is in flight passed the earlier
tombstone check but still persisted the fetched entry, resurrecting a
path whose remote delete is pending. Re-check right before CreateEntry.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: retract a lazily persisted entry when a delete raced the insert
The pre-insert tombstone check still leaves a window between the check
and the store insert. Since deletes always record the tombstone before
removing the entry, a tombstone visible right after a successful insert
means the delete already ran: delete the entry back out so the
tombstoned path stays deleted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: note why the replay deadline can safely open the gate
Deletes made after startup are captured by the live delete and event
paths, so a stalled replay can only be missing pre-restart deletes, all
of which are past the tombstone TTL by the deadline.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: retract only the entry a lazy remote read materialized
Deleting by path after a raced delete could remove a legitimate rewrite
that replaced the fetched entry. Verify the stored entry still matches
the remote object (or the just-created directory shape) before deleting,
and apply the same post-insert check to lazy listing children.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: require full-entry equality before retracting a lazy entry
Remote-only matching still removed a write that had updated the fetched
entry, e.g. appended chunks. Compare the persisted entry against what
this read materialized; any change means a real update owns the path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: VolumeMarkReadonly answers NotFound when the volume vanished under the lock
make_volume_readonly looked the volume up, notified the master (step 1),
then took the store write lock (step 2) and marked the volume only `if
let Some(..)`. When the volume left the store during step 1 -- a master
round trip, during which an unmount or a heartbeat expiry can land --
the missing else meant the RPC reported success for a volume the server
no longer has, and step 3 told the master again that it is read-only.
Go's Store.MarkVolumeReadonly (weed/storage/store.go) returns
"volume %d not found" when findVolume comes back nil, and
makeVolumeReadonly (weed/server/volume_grpc_admin.go) returns that error
before the step-3 notification. The Rust step 2 now does the same:
find_volume_mut(vid) -> Status::not_found("volume {vid} not found"), and
the `?` skips step 3, as it already did for a set_read_only_persist
failure. The scrub caller already matches NotFound to skip such a
volume instead of failing the whole report; it now actually gets it.
volume_mark_writable already returns NotFound under its write lock.
The regression test opens the step-1 window deterministically: step 1
awaits the current_master_url read lock, so the test holds its write
guard, lets make_volume_readonly park there after its own lookup
succeeded, unmounts the volume, then releases the guard. With no master
configured the notification is a no-op, so the write lock in step 2 is
the only place left that can notice the volume is gone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume server: trim comments on the vanished-volume mark-readonly path
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-25 22:05:53 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: refuse s3.bucket.create on an existing bucket
CreateEntry without o_excl replaces the bucket entry, dropping every
extended attribute: lifecycle configuration, owner, versioning and the
irreversible Object Lock flag. Send o_excl so a re-run fails with
'bucket already exists' instead of silently resetting the bucket.
* filer: fail exclusive creates when the lookup itself fails
CreateEntry discards FindEntry errors, so an o_excl create hitting a
transient store failure would take the insert path and upsert over the
entry it was meant to preserve. Propagate the lookup error when o_excl
is set; non-exclusive creates keep their existing semantics.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: test s3.bucket.create requests an exclusive create
Exercises the command end to end through a fake filer gRPC server and
asserts the OExcl flag reaches the wire along with the already-exists
error path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: synchronize captured requests and assert the exact bucket error
The fake filer records CreateEntry requests on the gRPC server goroutine,
so reads need the same mutex; the test also now checks for the exact
"bucket my-bucket already exists" message rather than any error that
mentions existence.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-25 22:04:54 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: validate ext and collection in gRPC CopyFile
Port the Go-side checks (checkVolumeFileExtension, checkVolumeCollection)
to the Rust volume server so a client-supplied collection or ext carrying
a separator or ".." cannot fold a path outside the volume directory.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: validate ext and collection in gRPC ReceiveFile
Same port on the write path: the file ReceiveFile creates is built from
client-supplied fields, so reject traversal there too. Reported through
the response error field, matching Go's SendAndClose.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: stamp entries with IF_CHUNKS_EQUAL so a stale write-back cannot delete live chunks
updateLocalEntry records the RemoteEntry stamp after an upload by writing the
event's entry back with UpdateEntry. The filer deletes every stored chunk
absent from an updated entry, so when the file was rewritten while its upload
was in flight (or the event is a replay), the stale snapshot deletes the
rewrite's chunks: the entry then points at the new fid with no needle behind
it, and the rewrite's own upload fails and is skipped as superseded.
The stamp write now carries WriteCondition IF_CHUNKS_EQUAL over the event's
chunk fids, evaluated by the filer under the path lock. A refused stamp means
the filer moved past this event; the superseding event follows in the log and
stamps the current entry, so the refusal is logged and skipped like a
superseded upload.
Reproduction: weed server -filer plus a weed server -s3 remote, remote.mount,
filer.remote.sync; hold the remote (docker pause) so one upload stays in
flight, rewrite the file through the filer, unpause. Before: the entry's chunk
is 404 on every volume server. After: the stale stamp is refused, the rewrite's
chunk stays live and reads back after a vacuum.
* filer.remote.sync: stamp entries with IF_ENTRY_EQUAL so stale inline content or metadata cannot be restored
The IF_CHUNKS_EQUAL guard compared only the chunk fid multiset, so a
rewrite that touched inline content or metadata alone still compared
equal and the stale snapshot overwrote the live entry. The new clause
compares the whole stored entry against the event's entry under the
same path lock.
* filer: route conditional UpdateEntry to the entry's owner filer
Two filers locking the same path locally could still pass a stale
condition on the non-owner while the owner's entry had moved on. When a
condition or expected_extended precondition is set, forward the request
to the entry's owner the same way conditional CreateEntry does, with
is_moved bounding the hop.
* filer: compare IF_ENTRY_EQUAL against the normalized expected entry
FindEntry grows FileSize to the chunk extent, so a raw event entry with
FileSize still zero failed the condition on an unchanged file and the
stamp was skipped, letting a replay upload the object again.
* filer.remote.sync: classify refused stamps by gRPC status only
A FailedPrecondition substring in an unrelated error would have been
swallowed as a skipped stamp; status.FromError already unwraps.
* remote sync: keep the event entry intact for IF_ENTRY_EQUAL
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
OLake (github.com/datazip-inc/olake) is a CDC and ingestion engine that writes
Apache Iceberg. It covers two paths none of the existing catalog suites reach.
It is a strict Java Iceberg client. OLake does not write Iceberg from Go — its
Go process spawns a Java sidecar over gRPC and writes through the official
Apache Iceberg library, because the Go library has no equality deletes and CDC
needs them. That makes it the client class weed/s3api/iceberg's metadata
compliance backfill exists to serve: the one that fails with "Cannot parse
missing long current-snapshot-id" when spec-required keys are omitted.
And it produces equality deletes. Its upsert path commits operation=overwrite
with an equality-delete file and a delete manifest. ClickHouse, Doris, Trino,
Spark and DuckDB all only append, so nothing else in this directory exercises
a delete manifest at all.
Six subtests: the destination check reaches SUCCEEDED and actually loads the
REST catalog; discover enumerates the source; a full sync commits a snapshot;
PyIceberg reads back what the Java writer committed; an update plus a re-sync
records an overwrite carrying equality deletes with a delete manifest in the
current snapshot; and the catalog does not rewrite manifests the official Java
writer produced, which gives the manifest-repair path a negative test to go
with the ClickHouse positive one.
What it deliberately does not assert is a delete-applied read. PyIceberg
refuses to scan a table carrying equality deletes (apache/iceberg#6568) while
reading its metadata fine, and an engine that can apply them costs a
multi-gigabyte image. Recording the commit correctly is the catalog's
contract; applying deletes on read is the engine's. The README says so, says
the read half was verified by hand once with StarRocks, and warns against
later "upgrading" this to a PyIceberg rows read — which would either fail or,
if PyIceberg ever starts skipping deletes instead of raising, pass by not
looking.
The workflow job asserts the suite actually ran rather than trusting a green
exit, for the same reason: at least one top-level PASS and zero SKIP.
No product change — the destination config is the generic catalog_type=rest
with the standard OAuth2 client-credentials flow, and s3_path_style is not even
set, since OLake turns it on itself whenever s3_endpoint is non-empty.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* filer: resolve the collection a bucket delete drops
A bucket delete dropped the collection named after the bucket, which
assumes bucket name is collection name. With a collection rule the
write path honors, deleting the bucket either orphaned its collection
or, when a bucket was named after a shared collection, removed volumes
other buckets still write to.
Resolve the collection through the same rule chain the write path uses
and drop it only when no other bucket resolves there too. A listing
failure keeps the collection, the safe side of an unknown.
* filer: prove collection exclusivity across all paths before dropping it
The sibling-bucket scan missed every non-bucket writer: a broad rule like
'/' or '/buckets/', a rule under a surviving bucket, or a rule on an
unrelated path can route into the same collection. Check every storage
rule's prefix instead, and mirror the grouped gateway's explicit
<group>_<bucket> collection, which otherwise resolves a rule-named
collection the bucket never wrote to.
* s3: let the filer own the collection decision on bucket delete
Both entry points deleted a name-derived collection around the filer's
own resolved delete, bypassing its exclusivity check and wiping sibling
data. The filer now resolves the collection a bucket actually used,
including the grouped form.
* filer: keep a collection the default write route also uses
Rule-less writes outside buckets land in the filer's default collection,
so a bucket resolving there shares it with them.
* topology: wake the vacuum dispatcher when a worker frees quota
The dispatch loop slept a fixed 10s whenever every pending volume was
waiting for a per-server quota slot, so a sweep took volumes x 10s
regardless of how fast the compactions were. Workers now signal on a
buffered channel after crediting quota; the dispatcher waits on it with
the 10s sleep kept only as a timeout.
* master: add -vacuumIntervalSeconds to tune the automatic sweep interval
The 14-minute base interval was a literal inside the refresh loop while
every neighbouring vacuum knob was already a flag. Defaults to 840s,
unchanged.
* topology: keep the 14 minute floor on the vacuum interval
A zero-valued MasterOption or a negative -vacuumIntervalSeconds left the
sweep sleeping only its jitter, so treat non-positive intervals as the
previous default.
The parity pass only queued shards past the per-type cap, so a single
parity shard sharing a rack with data was never a move candidate even
when an empty data-free rack existed (2+1 over 3 DCs settled 2/1/0).
Non-overflow candidates now move too, but only to a rack without data;
overflow shards keep the existing data-rack fallback.
* volume server: sweep stale EC artifacts before VolumeEcShardsGenerate re-encodes
The Rust VolumeEcShardsGenerate went straight into write_ec_files: no unload
of an already-mounted EC volume and no stale-artifact sweep. Only .ec00..ecNN
on the encoding disk were truncated, so a retry could mix two encode runs. A
stale N.ec03 left on a sibling disk survived, reconcile later mounted it
against the new .ecx, and the new .vif made the encode_ts_ns identity guard
pass, so reads served old-run bytes at new-run offsets.
Mirror Go's VolumeEcShardsGenerate (#9880 / #9953): UnloadEcVolume on every
disk, then removeStaleEcArtifacts on every disk location before encoding.
remove_ec_volume_files_full_teardown already has removeStaleEcArtifacts'
semantics (.ec00..ec31, .ecx/.ecj/.ecsum[.vN] in both the data and idx dirs,
.vif only on a shard-only disk; never the source .dat/.idx), so reuse it. Add
Store::unload_ec_volume, which unlike remove_ec_volume does not stop at the
first disk and closes the descriptors so the unlink frees the inodes. The
store write lock covers only unload + sweep, not the encode.
The failure arm now also drops the generation-0 .ecsum, as Go's defer does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume server: wake the heartbeat after VolumeEcShardsGenerate unloads shards
The pre-encode unload drops mounted EC shards from memory, but unlike every
other unmount path it did not wake the heartbeat, so the master kept routing
reads to shards this server no longer serves until the next pulse. Notify
once the store lock is released, and before the sweep error propagates: a
failed sweep has unloaded the shards too.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume server: clean up encode artifacts when the .vif write fails too
Go's shouldCleanup defer covers every error before the .vif commits,
not just a failed encode. A serialize or write failure on the .vif left
the fresh .ecNN/.ecx/.ecsum behind, which the next generate would have
to rely on the new sweep to remove. Extract the cleanup and run it on
the .vif error paths as well.
* volume server: write the EC .vif atomically
Go's SaveVolumeInfo writes a temp file, syncs it, and renames it over
the target, so a failed write leaves the previous metadata intact and a
read-only .vif fails the save. The direct fs::write truncated the file
first, so a write or sync failure could leave an empty .vif even after
cleanup_encode removed the generated shards.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: an EC volume needs a non-empty .ecx to mount
Two gaps against Go in how the Rust volume server treats the .ecx.
EcVolume::new mounted with no index at all. The per-shard
VolumeEcShardsMount path picks the disk by shard file alone, so a shard
whose .ecx was on no local directory still registered and was
advertised to the master; every VolumeEcShardRead then failed with
"ecx file not open", and add_shard's 0-byte guard was neutralised
because ecx_file_size stayed 0. Go's NewEcVolume returns an error
wrapping os.ErrNotExist. EcVolume::new now fails with NotFound, and
Store::mount_ec_shard looks up the .ecx owner across all disks first
(findEcxIdxDirForVolume) so a shard on a sibling disk of its index
still mounts instead of turning into a hard failure.
A 0-byte .ecx stub, as left by a failed EC distribute copy, counted as
a valid index. Go requires Size() > 0 wherever the file steers a
decision: HasEcxFileOnDisk, findEcxIdxDirForVolume, indexEcxOwners
(shared by reconcile and mirror), and VolumeEcShardsCopy removes a
copied 0-byte .ecx and fails the copy. Mirror each through one
is_usable_ecx_file helper. NewEcVolume itself still accepts a lone
0-byte .ecx as a legitimate empty index, but prefers a non-empty copy,
local directory first, over a stub in the other directory; the
resolution in EcVolume::new now follows the same order.
Tests that mounted EC volumes without any .ecx get a real fixture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: mount_ec_shard tries every disk; reconcile ignores a 0-byte local .ecx
mount_ec_shard returned the first disk's error, so an unusable shard copy
(a 0-byte .ecNN left by an interrupted move) hid a good copy on the next
disk. Like Go's MountEcShards, keep scanning: NotFound means "not this
disk", any other failure is collected, and an all-disks-fail error names
every disk tried. "No .ecx on any local disk" is now told apart from
"shard not on this server".
The orphan-shard reconcile took its locally-mirrored fast path whenever a
local .ecx existed at all. A 0-byte stub there registered the shards against
an empty index while the owner index skipped that same stub. Go gates the
fast path on HasEcxFileOnDisk; do the same. ec_local_ecx_path loses its last
production caller and becomes test-only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: match Go's mount error text and skip the owner stat on the owning disk
MountEcShards in Go skips the HasEcxFileOnDisk stat when the disk's own
directories already hold the .ecx, dedups a shared -dir.idx across
locations in findEcxIdxDirForVolume, and reports "load failures" with
the same wording. Also drop two issue-number references from comments.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
With SEAWEED_WRITE_QUEUE=1 every upload came back with ETag "00000000".
The upload handler built the needle with Needle::default(), so its
checksum was CRC(0), and handed a clone of it to the queue. The CRC was
only computed in the write path, on the worker's clone, and WriteResult
carries no checksum back, so n.etag() in the handler formatted the zero
checksum. The direct path writes through &mut n and was correct.
Compute the checksum in the handler while building the needle, the way
Go's CreateNeedleFromRequest does, over the same bytes the write path
hashes (the stored data, gzipped or not). The ETag and the has-name flag
are read before the write, so the needle is moved into the queue instead
of cloned, which also drops a full payload copy per queued upload.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* volume: walk_index_file keeps row alignment across short reads
walk_index_file issued one Read::read per batch and decoded whatever came
back. Read::read may legally return a short count that is not a multiple
of the 17-byte entry size (FUSE and network filesystems, a BufReader whose
capacity is not a multiple of 17). The split entry at the end of the batch
was dropped with no carry and the next read started mid-entry, so every
later row was decoded from misaligned bytes and fed to the index as a
garbage key/offset/size. This function backs every in-memory index load.
Go's WalkIndexFile is immune because it reads through io.ReaderAt, which
returns a full buffer or an error. Fill the batch buffer until it is full
or the reader reports EOF, retrying ErrorKind::Interrupted, and only then
decode whole entries. Reads stay batched at ROWS_TO_READ entries.
EOF semantics are unchanged and match Go: on io.EOF Go decodes the whole
entries in the final buffer, ignores a trailing partial entry and returns
nil. A torn final entry is still skipped without an error here.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: trim walk_index_file comments
The batch-fill loop and the ShortReader test helper each carried a
paragraph where a sentence suffices.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* lance: accept OAuth2 bearer tokens for catalog auth
Lance and LanceDB clients can only send OAuth2 / Bearer / API-Key
headers on catalog calls, never SigV4, so behind an auth-enabled S3
gateway every namespace request failed with 403 Access Denied.
Mirror the Iceberg catalog's OAuth2 support: POST /oauth/token accepts
an S3 access key / secret key as client_id / client_secret, validates
them against IAM, and returns a signed JWT. The Auth middleware accepts
that token as a Bearer credential before falling through to SigV4.
Closes#11430
* lance: accept x-api-key header carrying an S3 credential
The Lance namespace spec's third auth scheme maps api_key onto the
x-api-key header. Accept "access_key:secret_key" there and validate it
against IAM, so clients that only hold static headers can authenticate
without minting a token first.
* lance: answer invalid_client with the Basic challenge
RFC 6749 5.2 requires a 401 from the token endpoint to carry
WWW-Authenticate matching the scheme the client used, so it knows how
to retry.
* lance: cap the token endpoint request body
/oauth/token is unauthenticated, so ParseForm needs the same size
bound decodeBody applies to every other catalog request.
* lance: keep query strings out of request logs
/oauth/token rejects a client_secret sent in the query, but the
logging middleware and the catch-all wrote RequestURI to the log
before that rejection ran. Log the path alone so a mis-sent secret
never reaches the log.
* lance: log the escaped path, not the decoded one
URL.Path decodes percent escapes, so a request like /%0aFORGED could
split log lines. EscapedPath keeps the encoding while still dropping
the query string.
* fix 11400
* persist failed-recovery quarantine and harden rollback
- record the unavailable state in a .unavailable marker, fsync it, and
re-arm it on load so a restart cannot serve an unverified pair
- quarantine the volume so heartbeats stop advertising it
- block MarkVolumeWritable while unavailable, rechecked under noWriteLock
- fail every request of a failed batch, not only the succeeded ones
- restore the needle map and truncate .dat on inline fsync rollback failure
- add truncateIndex for the sorted-file needle map
- mirror the fail-closed semantics in the Rust volume server
* volume: erase rolled-back mappings instead of leaving tombstones
A rolled-back batch or failed inline write used Delete() to undo a
needle that did not exist beforehand, leaving a tombstoned map entry
whose stale offset makes the next write to that needle fail reading a
header that no longer exists. Add removeMapping/restoreMapping to the
mappers so recovery erases entries that were absent before the batch
and reinstates the exact prior offset/size for ones that were,
including tombstones. The index row still goes through Delete so a
replay forgets the needle.
* volume: gate bulk readers on unavailable and fsync the marker's dir
- fsync_dir(&self.dir) synced the volume dir's parent, not the dir
holding .unavailable; pass the marker path so the create survives
a host crash
- export UnavailableError and check it in ReadAllNeedles,
VolumeTailSender, VolumeIncrementalCopy, and IncrementalBackup so
replica-sync paths cannot stream or append data from an unverified
.dat/.idx pair; mirror on the Rust side via read_dat_slice,
read_all_needles, dat_scan_plan, and the incremental-copy handler
* volume: drop issue references from comments near touched code
* volume: stop active scans when the volume becomes unavailable
The stream entry-point checks ran once per RPC, so a volume quarantined
by a failed recovery mid-scan kept serving data. Recheck availability
per needle/chunk on the detached read paths: tail scan and heartbeat,
read-all, incremental copy, incremental backup writes, and the Rust
StreamingBody chunk reads. Rust incremental copy also rejects a
quarantined volume before sync_to_disk touches the backend.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust volume: test makeup_diff replay across a 32 GiB offset boundary
Issue #11410 corrupted a replayed write's index offset in Go's makeupDiff
by patching only four of the five offset bytes. The Rust makeup_diff
already encodes the whole offset through idx_entry_to_bytes and
Offset::from_actual_offset; this adds the mirror of
TestConcurrentWriteCrossesOffsetBoundary so a regression would fail here
the same way it does under -tags=5BytesOffset on the Go side.
Sparse-truncate the .dat to 64 GiB, compact, write, commit: the index
offset must equal the .cpd size and the needle must stay readable
through a second vacuum. Gated on the 5bytes feature since a 64 GiB
.dat exceeds the 32 GiB range of 4-byte offsets.
* rust volume: skip the offset-boundary replay test on Windows
Windows set_len allocates the full 64 GiB extension instead of a sparse
range, so the test fails with StorageFull on CI runners. Gate it to unix,
where set_len leaves the extension unallocated.
* volume: encode all offset bytes when makeupDiff replays a write
makeupDiff patched only bytes 8:12 of the index entry, so under the
5BytesOffset build the fifth byte kept the old offset's high bits and the
replayed needle's index pointed 32 GiB-aligned ranges away from its body.
A later vacuum then dropped the entry as unreadable. Rebuild the entry with
needle_map.ToBytes, the same encoder the tombstone branch just below uses.
* volume: test makeupDiff replay across a 32 GiB offset boundary
Sparse-file test: truncate the .dat to 64 GiB after one write, compact,
write a second needle, commit, and assert the index offset matches the
.compacted size and the needle stays readable through a second vacuum.
Only runs under -tags=5BytesOffset.
* volume server: reject non-ASCII input instead of panicking
Three parsers sliced attacker-supplied strings by byte offset, so a
multi-byte character split inside itself and panicked the task:
- parse_needle_id_cookie took the last 8 bytes as the cookie and the
rest as the needle id. Reachable from VolumeServer.BatchDelete,
whose file_ids come straight off the wire as protobuf strings;
that handler already answers 400 per bad fid, so the guard turns a
panicked RPC into the error it was already written to return.
- TTL::read took the unit as the last byte and the count as
everything before it, so "?ttl=5<multi-byte>" split mid-character.
The HTTP upload path does TTL::read(..).ok() and drops an invalid
TTL; AllocateVolume maps the Err to InvalidArgument.
Both now reject non-ASCII up front. Hex and a digits-plus-unit TTL are
ASCII by definition, so no accepted input changes -- covered by tests
alongside the rejection cases.
The six response-* header overrides were inserted with
parse().unwrap(). They come from the query string, so
"?response-cache-control=%0Aevil" decodes to a value HeaderValue
rejects and the unwrap panicked the connection task,
unauthenticated. They now skip the override, matching the if-let the
chunked-response path in the same file already uses.
ReplicaPlacement::from_string was reported as a fourth site but is not
one: reaching chars[2] requires chars[0] and chars[1] to be ASCII
digits, which forces the padded string to be three single-byte
characters, so a multi-byte character always lands on a to_digit()
None first. Kept as a regression test rather than a change.
Each fix was confirmed against the unfixed code first: the parser
tests panic with "byte index N is not a char boundary", and the
integration tests panic at handlers.rs:1413 and ttl.rs:88.
Not a vector, contrary to the report: the HTTP request line. The path
is not percent-decoded before parsing, so "%C3%A9" stays ASCII and
fails the length check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* volume server: fall back to needle MIME when response-content-type is invalid
Skipping an unparseable override left the response without any
Content-Type because the override had already bypassed the normal MIME
selection. Also correct a test comment that described a chars[2] panic
which cannot be reached.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* s3: count seaweedfs-quota as an operation subresource
PUT /bucket?policy&seaweedfs-quota was not rejected by
hasAmbiguousSubresource because operationSubresources omitted the
seaweedfs-quota key. The router then picks the policy route
(registered first) while the IAM action resolver may resolve the
request to s3:PutBucketQuota, letting a quota-only identity write a
bucket policy. Reject the combination before routing, matching the
fix for policy&tagging (#10987).
* s3: resolve seaweedfs-quota after other bucket subresources
The quota routes are registered last among the bucket subresource
routes, but the action resolver found seaweedfs-quota inside the
unordered bucketQueryActions map, so a request carrying it alongside
another selector could be authorized as the quota operation while the
router served the earlier-registered handler. Resolve it explicitly at
the end so the resolver agrees with the router, mirroring how
list-type is handled.
* s3: count resolver subresources in the ambiguity guard
hasAmbiguousSubresource only counted operationSubresources, so adding
a query parameter to the action resolver without updating that list
reopened the authorize-one-serve-another gap. Count bucketQueryActions
keys as operation selectors too, and add a test that walks the
registered routes and fails on any query key that is neither an
operation subresource nor a known modifier.
* volume: load the .ecj deletion journal in chunks, and repair a torn tail
Two independent defects in the EC deletion journal's load path.
1. The loader issued one NEEDLE_ID_SIZE-byte positional read per entry.
That is fine for a healthy journal -- kilobytes -- and pathological for a
large one. A `.ecj` is semantically a SET of deleted needle ids but is
written as an append-only log that nothing dedupes, and several paths append
a peer's ENTIRE journal onto the local one (VolumeEcShardsCopy with
copy_ecj_file, EC index recovery, and ec_decode's deliberate cross-holder
merge), so a volume whose shards are repeatedly balanced between two servers
grows the file without bound.
Observed in production: 1.51 TB and 1.30 TB on the two holders of one 10+4
volume containing ~100 distinct ids. At that size the per-entry loop is
~188e9 syscalls, run synchronously while holding the deleted_needles write
lock and before the HTTP port opens. The process sits at 100% of one core
with a small RSS -- the set stays tiny because the ids repeat -- reading at a
few MiB/s because 8-byte reads defeat readahead, logs nothing after "Adding
storage location", and ignores SIGTERM. The master then unregisters every
volume it holds and reads of them fail. 4.46 and 4.47 are both affected.
Read in 1 MiB chunks and build into a local set, merging once at the end so
the write lock is not held for the whole scan. Measured on a 256 MiB journal
of 100 distinct ids: 33,554,500 syscalls -> 257, identical resulting set.
2. A torn tail silently corrupted later deletes.
The journal handle is in append mode, so writes land at the physical end
regardless of alignment. A trailing partial record therefore pushed every
later append out of alignment: the loader skipped the partial bytes, but the
next mount decoded them together with the leading bytes of the following
entry, producing one garbage id and dropping the delete that came after the
tear -- after acknowledging it.
Truncate to a whole number of records at mount, before anything can append.
The repair uses its own read+write (non-append) handle: on Windows,
append(true) requests FILE_APPEND_DATA without FILE_WRITE_DATA (and
.write(true) is subsumed by .append(true)), so SetEndOfFile through the
journal handle fails with ERROR_ACCESS_DENIED.
The same trap exists in journal_delete's recovery path, which calls set_len
on the append handle to roll back a partial write whose sync failed. It is
error-handled rather than fatal, so on Windows that rollback silently does
not happen. Untouched here; worth a separate fix.
Bounding the journal's growth needs compaction, which is deliberately not in
this change: replacing the file under a store that can hold several EcVolume
instances for one volume id requires coordinating with the other holders, and
that belongs at the store layer. Sent separately.
Tests: a journal spanning several read chunks loads every entry; a trailing
partial record is ignored rather than panicking; a torn tail is truncated at
mount and a delete taken afterwards survives a remount.
* volume: roll back a failed .ecj append through a dedicated write handle
The append handle lacks FILE_WRITE_DATA on Windows, so the set_len
rollback after a failed sync silently did nothing and the journal could
drift one record past deleted_needles. Same trap as the torn-tail repair
in this file; fix it the same way. Also format the new tests.
* volume: mirror chunked .ecj load and torn-tail repair in Go
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
* volume server: ReceiveFile loses bytes and hides fsync failures
Three defects in one handler, all on the path that receives a pushed
.dat/.idx/.vif or EC shard:
- `f.write(&content)` never compared the return to content.len().
A short write (ENOSPC, NFS) counted only the bytes that landed,
so every later chunk was written at a shifted offset and the RPC
answered error: "" with a byte count that looked right. Go's
os.File.Write loops. Now write_all.
- `let _ = f.sync_all();` discarded EIO and answered success with
the full byte count. Go omits the check too, but
ReceiveFileResponse carries an `error` field and the caller
renames the staged file into place on success -- so a silent
fsync failure publishes a file whose data never reached the
platter. Flush and fsync failures are now reported.
- Both the per-chunk write and the final fsync were blocking
std::fs calls inside the async fn, on the runtime worker that is
also driving the stream. Switched to tokio::fs + BufWriter, the
shape `drain_copy_stream_to_file` in this same file already uses
and documents. The partial-file cleanup on the error path moves
to tokio::fs::remove_file for the same reason.
The handler had no test at all, which is how the short-write bug
survived. Added a round-trip over a real connection with ragged chunk
boundaries, asserting the bytes on disk and not only the reported
count -- a dropped or reordered chunk changes the file even when
bytes_written still adds up.
That test guards the rewrite; it does not reproduce the original
faults. ENOSPC and EIO need fault injection that this suite has no
harness for, so the short-write and fsync paths are argued from the
code, not demonstrated by a failing test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* volume server: remove the staged file on every ReceiveFile error reply
Flush and fsync failures returned early and left the partial .copying or
shard file behind, as did the pre-existing write-error path. Route all
response-level errors through one cleanup block, matching Go's
close-and-remove on a failed write.
* volume server: tighten ReceiveFile comments
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
* volume server: HTTP DELETE on a distributed EC volume
The delete handler validated the cookie with EcVolume::read_ec_shard_needle,
which reads only locally-mounted shards and errors "ec shard N not available
locally" for any interval held by a peer. Every Err was mapped to 500 and no
.ecj tombstone was appended, so on a standard 10+4 spread over 14 servers an
HTTP delete of an EC needle could not succeed. The GET path already goes
through read_ec_shard_needle_distributed.
Route the delete's read through the same distributed reader. It does a
local-first pass in its snapshot phase, so the all-shards-local case costs
what it did before, and no store guard is held across the await (the reader
takes its own; RwLockReadGuard is !Send).
Two smaller corrections fall out of the new return type:
- the reader reports both "needle not in the index" and "volume vanished
between the has_ec check and the snapshot" as Ok(None), which collapses
the old Some(Ok(None)) and None arms into one 404;
- an io::ErrorKind::NotFound now answers 404 rather than 500, matching the
GET path. Telling a caller to retry a delete that can never succeed was
half the bug.
The cookie check and its ordering before the journal append are unchanged.
Not addressed here: Rust journals the tombstone locally while Go routes it to
the primary shard holder. That is a separate behaviour change and belongs in
its own PR against the same issue-10 checkbox.
The regression test mounts 13 of 14 shards, leaving out the one holding the
needle's interval. The distributed reader seeds its Reed-Solomon buffers from
locally mounted siblings, so with >= 10 survivors it reconstructs with no peer
fan-out -- which makes the bug reproducible on a single node. Against the
unfixed handler the test fails with 500 vs 202.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* volume server: fail the delete when the EC volume unmounts mid-request
find_ec_volume_mut returning None used to fall through to a 202 with no
.ecj tombstone written, reporting success for a delete that did not
happen. Answer 404 like the other volume-vanished arms so the caller can
retry after a remount.
* volume server: forward EC needle deletes to a primary-shard holder
Mirror Go's doDeleteNeedleFromAtLeastOneRemoteEcShards: the tombstone is
journaled on one holder of the needle's primary data shard via
VolumeEcBlobDelete (or the local journal when this server holds the
shard), falling back to any other shard holder when the primary has
none. Journaling only on the node that received the DELETE scattered
tombstones across whichever server took the request.
* volume server: route BatchDelete EC deletes through the same forwarding
BatchDelete had the same local-journal divergence as HTTP DELETE, plus a
gap the old code admitted in a comment: the .ecx index cannot supply the
needle's cookie, so EC deletes ran with no cookie check at all. A
distributed read now fills the needle for every EC entry — matching Go's
DeleteEcShardNeedle, which reads and compares the fid cookie even when
skip_cookie_check is set — and the tombstone forwards via
delete_ec_shard_needle_distributed. A needle deleted between read and
journal reports 304 like Go's ErrorDeleted; a vanished volume reports
500 so the filer retries.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
server/grpc_client.rs stopped at build_grpc_endpoint() -> Endpoint, so all 13
production call sites hand-wrote the same .connect() + X::with_interceptor()
+ two max_*_message_size() lines. Four of them -- VolumeCopy,
VolumeTailReceiver, VolumeEcShardsCopy and the HTTP chunk batch-delete fan-out
-- dialed with no timeout at all, so an unreachable peer whose TCP handshake
never completes (SYN dropped, blackholed route, host behind a silent firewall)
left the operation waiting on the kernel's own retry budget, minutes long.
Add GrpcDialOptions (unary / long / stream presets), connect_channel(), and
volume_server_client() / master_client() / filer_client() constructors that
attach the request-id interceptor and lift both message-size limits, then
route all 13 sites through them. build_grpc_endpoint is private again, so
connect_channel is the only way out of the module and no call site can dial
without picking up a bound. Each site's existing timeouts are preserved
exactly; the four bare dials gain a 5 s connect timeout and nothing else. No
per-request deadline was added to any streaming call: Endpoint::timeout is a
per-request bound on time-to-first-response-headers for every request the
channel carries, so a value picked for one short call would also be the header
deadline for the whole-volume transfer sharing the dial.
The new bound covers the TCP handshake only -- tonic hands connect_timeout to
HttpConnector::set_connect_timeout. A peer that completes the handshake and
then stalls in the TLS or HTTP/2 exchange is still unbounded at those four
sites, as are the RPCs themselves. That is why the three ping_* helpers keep
their outer tokio::time::timeout: replacing it with connect_timeout would have
narrowed a whole-connect bound they already had.
main.rs no longer re-declares GRPC_MAX_MESSAGE_SIZE and the three
keepalive/window constants; it imports them from grpc_client.rs so the
inbound server and the outgoing clients cannot drift apart.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: share the I/O-error tracker between Volume and EcVolume
Volume and EcVolume each carried the same three fields - a mutex-held
last error, a consecutive count and a sticky quarantine flag - and the
same four methods over them, identical except for the path qualifier on
is_storage_io_error. The tolerance the count is compared against was a
fourth copy: heartbeat.rs held VOLUME_IO_ERROR_TOLERANCE for volumes,
ec_volume.rs held IO_ERROR_TOLERANCE for EC, and the volume test helper
open-coded the same 3, so the two paths could drift apart silently.
Go keeps this in one place already: weed/storage/io_error.go holds
IoErrorTracker, IoErrorTolerance and isStorageIoError, and Volume embeds
the tracker. Go's EcVolume has to re-implement it only because those
fields are unexported and EC lives in another package.
storage::io_error::IoErrorTracker now owns that state, with record /
state / should_quarantine / mark_quarantined / reset and the single
IO_ERROR_TOLERANCE. is_storage_io_error moves into the same file, so it
sits with the tracker that is now its only caller, the way io_error.go
is laid out. Both volume kinds embed one tracker and keep their existing
method names as delegates, so the ~16 internal call sites and the
readers in heartbeat.rs, store.rs and grpc_server.rs change only where
the two threshold comparisons become should_quarantine().
Volume::last_io_error and EcVolume::reset_io_error_state had no callers
and are gone.
Unchanged: what counts as a storage-media error - is_storage_io_error
changed file, not body, and is still the single predicate both volume
kinds share, where Go's EcVolume tests EIO directly and so misses the
Windows codes. Also unchanged: the tolerance value, the metric increment
on every counted error, and the sticky quarantine - a success clears the
count and the last error but never the flag, which only reset lifts. In
the heartbeat the state read moved inside the quarantine branch, so the
common path no longer takes the tracker's mutex or clones the last-error
string; should_quarantine's two relaxed loads run either way.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: hoist absolute_display_path into server
handlers.rs and ui.rs each held a byte-identical copy of the helper that
turns a configured -dir into an absolute path for display. The status
JSON and the status page are meant to show the same directory, so the
two copies had to be edited together to stay that way.
The helper now lives in server/mod.rs as pub(crate) and both callers use
it. No behaviour change: same body, same call sites.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: keep EcVolume::reset_io_error_state
Moving both volume types onto the shared IoErrorTracker dropped
EcVolume's public reset while Volume kept its own, so the two sides of
the tracker drifted apart.
mark_quarantined is sticky: a later successful read clears the error
count through record(), but the quarantine flag only comes down through
reset(). Without the delegate an EC volume that hit sustained media
errors could not be returned to service in place once the storage was
repaired. Go exposes the same method as EcVolume.ResetIoErrorState
(weed/storage/erasure_coding/ec_volume.go:114).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* rust volume: name the shared tracker after Go's IoErrorTracker
- check_read_write_error, get_io_error_state, mark_io_quarantined,
reset_io_error_state match weed/storage/io_error.go one to one
- io_error module is pub(crate) like the io module beside it
- restore EcVolume::reset_io_error_state so both volume kinds expose the
same recovery surface
- trim comments that restate the code
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust: a seaweed-common crate for the address and TLS helpers both crates carry
seaweed-volume and seaweed-worker are separate cargo trees with separate
lockfiles and no root manifest, so anything both of them need has had to be
written twice. Two of those copies are a correctness risk rather than a typing
cost, and this crate is where they stop being copies.
address.rs is the HTTP<->gRPC port rule: `host:port` means gRPC on port+10000,
`host:port.grpcPort` names it outright. The two copies had already drifted —
the worker's bracketed IPv6 literals, the volume server's did not — so the rule
lives here once, returning a typed AddressError whose Display text is the volume
server's original wording, with join_host_port public beside it. A test asserts
two of those messages in full rather than by substring, because the wording is
the contract its callers hand to a Status or an io::Error; the other three end
in a std ParseIntError message, which is std's to reword. The enum is
#[non_exhaustive] so a future variant is not a breaking change for either
consumer. The tests are both crates' cases together, plus the IPv6,
already-bracketed and normalisation cases neither copy covered on its own.
tls.rs is install_default_crypto_provider. Both binaries link aws-lc-rs and ring
transitively, so rustls cannot auto-select and tonic's client TLS panics on
first use; each binary has to pin one and it has to be the same one, which is
exactly the kind of choice that should not exist twice. It is safe to share
because `cargo tree -i rustls` resolves a single rustls in each tree (0.23.37 in
seaweed-volume, 0.23.43 in seaweed-worker) and cargo unifies all
semver-compatible `rustls = "0.23"` requirements into one crate per binary, so
this crate writes the same process-wide static its consumer reads. rustls is
already in both graphs — directly in the volume server, through tonic's
tls-aws-lc in seaweed-worker-core — so the dependency adds no crate to either.
rust-version is 1.91.1, the lower of the two consumers' floors, so depending on
this crate cannot raise either tree's MSRV; verified with
`cargo +1.91.1 check --all-targets`. The lockfile is committed even though this
is a library: CI builds it directly, so a committed lock is what makes those
runs reproducible and their caches stable.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust: take the address and TLS helpers from seaweed-common
Both public signatures are kept, so no caller outside the two wrapper files
changes. parse_grpc_address stays `Result<String, String>` and maps the typed
error through Display; server_to_grpc_address stays `Option<String>` and drops
it with .ok(). Their doc comments and the volume server's 13 call sites are
otherwise untouched.
Three behaviours change, each in the direction of the copy that was already
right:
- The volume server now brackets IPv6 literals. `::1:19333` used to come back as
`::1:29333`, which build_grpc_endpoint rejects with "invalid gRPC endpoint
http://::1:19333: invalid authority" — an IPv6 master or EC peer could not be
dialled at all. Two tests in grpc_client.rs pin it, one on the string and one
on the endpoint the string builds.
- The volume server now emits the *parsed* gRPC port of the dotted form instead
of the original text it had just validated, so `host:8080.018080` and
`host:8080.+18080` come back as `host:18080` rather than as authorities the
URI parser rejects. Same port either way; only malformed spellings change.
- The worker's dotted form now validates the HTTP port it discards.
`server_to_grpc_address("host:abc.18080")` used to answer Some("host:18080");
it now answers None, which is what the volume server's copy has always done.
install_default_crypto_provider becomes a re-export in both trees, so
`crate::security::tls::install_default_crypto_provider` and
`weed_lance_worker::tls::install_default_crypto_provider` still resolve. The
lance crate's `rustls = "0.23"` was its only direct use of rustls and goes away
with the body; seaweed-common states the same requirement, so neither the
resolved version nor the enabled features move in either lockfile.
The PEM test fixtures stay where they are. The two tests that use them are not
duplicates: the volume server's exercises build_grpc_endpoint, and the lance one
exists precisely because aws-lc-rs and ring are both linked in that crate's
graph. Only the literals are shared, and exporting test fixtures from a library
to dedupe two constants costs more than it saves.
A path dependency outside both trees means every build context that copies one
crate directory has to copy the other. The repo has one: the Rust source-build
stage of docker/Dockerfile.go_build, which now copies seaweed-common beside
seaweed-volume. Every workflow whose `paths:` filter keys on a crate directory
gains `seaweed-common/**` — the two Rust test workflows, rust_binaries_dev,
container_dev and performance. The tag- and dispatch-triggered ones
(rust_binaries_release, container_release_unified, container_latest) have no
`paths:` filter and need nothing.
The two Rust test workflows also run `cargo test` in seaweed-common, from their
unit-test job, because a path dependency is not a workspace member and neither
tree's own `cargo test` reaches it. Each step builds into its job's cached
target directory, and both cache keys now hash seaweed-common/Cargo.lock as well
so a change there invalidates the cache it would otherwise silently reuse.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docker: keep go_build working for BRANCH revisions without seaweed-common
The rust_builder stage copies seaweed-common unconditionally now that seaweed-volume path-depends on it, but BRANCH can name any revision — including ones that predate the crate. Create the directory in the builder stage so the COPY always has a source; an empty dir beside an old seaweed-volume is harmless.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust volume: one S3 tier registry instead of two kept in sync by hand
`VolumeServerState.s3_tier_registry` and `global_s3_tier_registry()` held
the same S3 tier backends. `apply_storage_backends` — the only production
writer — registered every backend into both, and each half of the tiering
code then read a different one: the gRPC tier-move handlers resolved the
backend from the per-server field, while `Volume`'s remote mount and
destroy paths resolved it from the global registry, because a `Volume` has
no handle to the server state. Two registries that must agree, kept in
agreement by a duplicated `register_s3_backend` call and a comment in a
test constructor explaining the hand-sync.
Delete the field and let both tier-move handlers resolve from the global
registry, so `apply_storage_backends` registers once and no longer needs
the server state at all. Injecting a registry handle through `VolumeSpec`
instead was considered and rejected here: it would touch every `Volume`
constructor for no functional gain, and the process-wide registry is what
`Volume` already uses.
Behaviour is unchanged: the same names were registered in both registries,
so every lookup resolves exactly as before. The tier-down test now
registers its backend only in the global registry — before this change it
fails with `remote storage s3.tier_down_delete not found from supported:
[]`. The tier-up handler had no test at all, so it gets a cheap probe:
register a backend only in the global registry, ask for that destination,
and check the call gets past the lookup — the response is dropped straight
away, so the transfer sees a departed caller and never opens a connection.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: await the tier-up probe terminal error instead of racing it
Dropping the response left it to chance whether the detached transfer saw the closed channel before its initial check; if it won that race it went on to attempt the multipart upload with no one waiting on the outcome. Hold the stream and read until the dead endpoint fails the upload — the terminal error proves the task ran and finished, so no background network work outlives the test.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The per-EcVolume shard-location cache was three fields under three locks:
an RwLock<HashMap> for the map, a Mutex<Option<Instant>> for the time it
was last refreshed, and a Mutex<bool> for the stale mark. Nothing tied
them together. merge_shard_locations published the merged map, released
the write lock, and only then stamped the refresh time; both readers
(scrub_ec_volume_distributed's snapshot and build_snapshot) took the two
guards one after the other. A reader landing between the two writes
paired a freshly merged map with the previous lookup's timestamp -- and
that pair is exactly what needs_refresh judges, so a read went back to
the master for a map that had just been refreshed. Go keeps the same
state in one struct behind one ShardLocationsLock. replace_shard_locations
documented itself as "a single observable step" while being two.
Fold the three fields into one ShardLocationCache behind a single RwLock.
merge_shard_locations upserts and stamps in one write section,
shard_locations_snapshot returns the map and its time from one read
section, and mark_shard_locations_stale / claim_shard_locations_refresh
move the mark's read-and-consume onto the cache. The three zero-caller
accessors -- set_shard_locations, replace_shard_locations,
get_shard_locations -- are deleted, and the field is now private, so the
invariant cannot be sidestepped from outside the module. The two test
seeding sites go through merge_shard_locations, which already produces
the state they were writing by hand.
Unchanged: the freshness rule. needs_refresh keeps its thresholds and
still judges the caller's snapshot -- the map that caller will actually
read from, not whatever is cached by the time the claim runs -- so only
the stale mark is read from under the new lock. The master lookup, the
completeness guard in write_back_shard_locations and the per-shard upsert
semantics are untouched.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: typed errors for store compaction so gRPC can answer NotFound
The vacuum entry points on `Store` returned `Result<_, String>`, so the
gRPC layer had nothing to branch on and answered `Status::internal` for
every failure. A vacuum loop that races a volume being moved or deleted
saw the same code as a disk going bad, and `weed shell` could only tell
the two apart by matching on the message text.
`VolumeError` gains `VolumeNotFound(VolumeId)` — the existing `NotFound`
is needle-level and carries no payload — and `InsufficientSpace`, and
`compact_volume`, `commit_compact_volume`, `cleanup_compact_volume` and
`delete_collection` return it. `impl From<VolumeError> for tonic::Status`
in `server/mod.rs` maps not-found to `not_found`, read-only to
`failed_precondition`, insufficient space to `resource_exhausted`,
already-exists to `already_exists`, and everything else to `internal`;
the four RPCs prefix their own context with `status_with_context`, so a
message reads "commit compact volume 7: volume id 7 is not found". The
store-side "during compact" / "during commit compact" / "during cleaning
up" suffixes are gone, and the free-space message drops the volume id the
prefix already supplies.
`check_compact_volume` had no callers — `VacuumVolumeCheck` computes the
garbage level from its own `find_volume` — and is deleted. `compact_volume`
folded the size estimate into its first lookup, dropping the `unwrap()`
re-lookup that only existed to dodge a borrow.
`ascending_visit` on `CompactNeedleMap`, `RedbNeedleMap`,
`SortedFileNeedleMap` and the `NeedleMap` dispatch is now generic over the
visitor's error type, like `CompactMap::ascending_visit` already was. The
three signatures that can fail on their own bound `E: From<String>` to
carry those failures; the in-memory walk in `iter_entries` names
`Infallible`, which says in the type what its comment used to say in prose.
No Go shell command matches on the old error text: the strings exist only
in weed/storage/store_vacuum.go.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume server: trim comments and answer the same codes from Go
- vacuum_volume_check reports VolumeError::VolumeNotFound like the other
vacuum RPCs instead of its own "not found volume id" wording
- drop doc comments that restate what the code says
- Go volume server wraps ErrVolumeNotFound/ErrInsufficientSpace from
store_vacuum.go so VacuumVolumeCheck/Compact/Commit/Cleanup and
DeleteCollection answer NotFound/ResourceExhausted, matching the Rust
volume server; volumeDeleteStatusError generalized to volumeStatusError
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: prefix operation context on vacuum errors
Lower-level errors forwarded by CompactVolume, CommitCompactVolume,
CommitCleanupVolume and DeleteCollection carry no volume id or operation
name. Wrap with %w so the status mapping still sees the sentinel chain,
matching the context the Rust server's status_with_context adds.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: map NotEmpty to FailedPrecondition, share mapper in VolumeDelete
Go's volumeStatusError maps ErrVolumeNotEmpty to FailedPrecondition; the
Rust Status conversion was missing it and volume_delete kept a hand-rolled
match. Route it through status_with_context like the vacuum handlers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Volume carried `pub has_remote_file: bool` next to `pub volume_info`,
and the bool was only ever the answer to `!volume_info.files.is_empty()`:
outside the two constructors, `refresh_remote_write_mode` was the single
writer. Both fields being public made the pair a convention rather than
an invariant. Every caller that touched `volume_info.files` — load_vif
twice, the tier-up handler, the tier-down handler and its rollback — had
to remember to call `refresh_remote_write_mode` afterwards, and a caller
that forgot would leave the volume advertising a write mode its .vif
contradicts, or serving a remote .dat through a writable needle map.
The bool becomes `has_remote_file()`, computed from the list, so it
cannot drift. `volume_info` becomes private with a `volume_info()`
reader, and edits to the reference list go through
`update_remote_files(|files| ...)`, which applies the closure and then
refreshes the derived write mode and the needle map. With no caller left
outside the module, `refresh_remote_write_mode` is private.
Unchanged: the refresh logic itself, the order of operations in both
tier handlers, and the tier-down rollback semantics. The rollback still
snapshots the removed reference before the refresh runs, restores it on
failure, and re-refreshes unconditionally on the error path — the second
`update_remote_files` call runs with a no-op closure when there was
nothing to restore, exactly as the old code re-ran the refresh whether
or not it had re-inserted a reference.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Three production `unsafe` blocks carried no `// SAFETY:` comment at all
(`libc::fallocate`, `libc::sysinfo`, `libc::statvfs`), and nothing made
that an error: `clippy::undocumented_unsafe_blocks` is a `restriction`
lint, allow-by-default, and appeared nowhere in either crate. Turn it on
in `seaweed-volume`'s `[lints.clippy]` and in the worker workspace's
`[workspace.lints.clippy]`, then document what each block relies on.
`memory_status.rs` and `disk_location.rs` get their blocks narrowed to
the `zeroed()` and the libc call, so each comment sits next to the
operation it justifies and the arithmetic is outside the block. Both
turn the success test into an early return on failure; the casts, the
multiplication order and the values returned on either path are
unchanged.
The bigger problem was in `config.rs`'s tests. `with_temp_env_var` and
`with_cleared_security_env` called `std::env::set_var`/`remove_var`,
claiming soundness because every caller holds `process_state_lock()`.
That mutex only serialises the fourteen annotated tests in this module.
The same lib test binary runs the `grpc_server.rs` tests, which bind a
`TcpListener`, dial loopback and drive a multi-thread tokio runtime, and
tonic/hyper/rustls/aws-sdk all read the environment lazily on those
threads — which is exactly the race Rust 2024 made these calls unsafe
for. `restore_env_var` had no SAFETY comment at all. `#[serial]` would
not have helped: it serialises annotated tests, which the mutex already
did.
So the config layer no longer reads the environment implicitly. An
`EnvLookup<'a> = &'a dyn Fn(&str) -> Option<OsString>` is threaded from
the public entry points down to every reader — `HOME`, `USERPROFILE`,
the twenty-four `WEED_*` keys and `SEAWEED_WRITE_QUEUE`. `parse_cli` and
`parse_security_config` keep their signatures and pass `process_env`, a
thin wrapper over `std::env::var_os`; `resolve_config` becomes
`resolve_config_with_env` (private, one caller). Tests build one with
`fake_env` instead, so no test touches the real environment and every
`unsafe` in the module is gone.
`process_state_lock()` stays, with a smaller job: `set_current_dir` is
safe but still process-global, so the tests that move the working
directory are still serialised against the ones that read it. Tests
naming an explicit config file never reach that search and no longer
take the lock.
No production behaviour changes: the same keys are read in the same
order with the same precedence, and `env_string` reproduces
`std::env::var(key).ok()` — absent and non-UTF-8 both read as unset.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
- README and values.yaml now describe the allowInsecureBind escape
hatch alongside the non-loopback bind guard
- remove a PR reference from the CI test comment
PR #11185 made `weed admin` refuse to bind a non-loopback address
without -adminPassword or mTLS. PR #11228 added -allowInsecureNoAuth
as an explicit opt-out for operators who restrict admin access some
other way (e.g. a NetworkPolicy plus an authenticating reverse proxy).
The chart's render-time guard added by #11236 (admin-statefulset.yaml,
seaweedfs.admin.authEnabled) predates -allowInsecureNoAuth and only
recognizes password-based auth, so there was no values.yaml path to
express that choice: the chart would fail(...) even though the binary
itself would start fine with a warning.
Add admin.allowInsecureBind (default false) to the seaweedfs.admin.authEnabled
helper's checks; when true it renders -allowInsecureNoAuth on the admin
command and satisfies the render guard alongside the existing
password-based checks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Presigned HeadObject/GetObject requests hoist x-amz-checksum-mode into the
signed query string, so a strict header-only check would withhold stored
checksums on presigned reads that AWS honors.
Presigned CompleteMultipartUpload requests hoist x-amz-checksum-type and
the full-object checksum header into the signed query string, so a
header-only lookup would skip BadDigest validation for them.
COMPOSITE uploads must still carry every part checksum in the complete
request, but FULL_OBJECT uploads may instead supply the whole-object
checksum in an x-amz-checksum-* request header. Compare that header
against the computed object checksum and return BadDigest on mismatch,
matching AWS.
An UploadPart that explicitly selects a different checksum algorithm than
the one declared at CreateMultipartUpload would store a checksum
CompleteMultipartUpload could never accept. Reject the conflict up front
with InvalidRequest, matching AWS.
Parse the Checksum* elements of each completed part and enforce what AWS
does for uploads created with x-amz-checksum-algorithm: every part must
carry a checksum in the complete request (InvalidRequest when missing,
BadDigest when it differs from the stored part checksum), and an
x-amz-checksum-type header must match the upload resolved checksum type
(BadDigest). Add the issue-11401 reproduction as a regression test.
AWS computes a checksum for every part of an upload created with
x-amz-checksum-algorithm, even when the part request carries no checksum
headers. Mirror that: when the part request specifies no algorithm, apply
the one stored on the upload entry so the part entry keeps a checksum
CompleteMultipartUpload can fold into the object checksum.
fix(volume): stop scans at a header they cannot advance past
A corrupt .dat header with a very negative size gives a record length
(NeedleHeaderSize + NeedleBodyLength) of zero or less: v3 sizes -43..-36
and v2 sizes -35..-28 give exactly zero, and smaller sizes give a
negative length. ScanVolumeFileFrom advanced by that length, so it
re-read the same header forever or stepped back into the record before
it. weed fix, weed export, weed compact, incremental weed backup and the
tail sender behind volume.move and volume.merge could hang on such a
volume, and weed compact could also finish with a .cpx that had dropped
every needle after the header.
Return an error wrapping needle.ErrorCorrupted instead. The check runs
after the visitor has seen the record, so the rebuild scanner still
stops quietly with io.EOF. Smaller negative sizes whose record length is
positive are still stepped over, preserving the salvage behavior
compaction relies on.
Mirror the guard into the Rust volume scans: DatScanPlan::scan and
read_all_needles fail on a non-positive record length, as does
scan_dat_head, so a corrupt header cannot stall a tail pass or leave the
repair scan walking stale offsets.
* fix(volume): reject negative sizes in ReadNeedleBlob and WriteNeedleBlob
A ReadNeedleBlob RPC with a size of -44 or below (-36 on v2 volumes)
panics in makeslice inside needle.ReadNeedleBlob. The volume gRPC server
has no recovery interceptor, so one request kills the process. Smaller
negative sizes return bytes that are not a record.
WriteNeedleBlob accepted a negative size whenever the blob header
carried the same value: it appended the blob to .dat and indexed the
needle with that size, which reads as deleted.
Reject size < 0 in both Volume methods. Size 0 still passes, since
delete records carry it. The Rust volume server got the same storage
guards in #11345.
* fix(volume): reject needle blobs whose length does not match their size
WriteNeedleBlob appends the blob as is. A blob that is not the length
its size implies leaves .dat off the 8-byte grid, and every later
ordinary write to the volume is indexed at a truncated offset and reads
back as EOF. A blob off by 8 bytes keeps the grid but leaves bytes that
a .dat scan reads as the next record.
The in-tree callers already send exact lengths. The one case this newly
refuses is a copy between volumes of different needle versions, and
that case already writes a broken record: a v3 record lands on a v2
volume with 8 extra bytes, and a v2 record on a v3 volume either fails
the timestamp check or lands 8 bytes short.
This is separate from the negative-size guards, whose Rust counterpart
is #11345. The Rust server does not check the length yet.
* fix(volume): guard the blob buffer allocation in needle.ReadNeedleBlob
Volume.ReadNeedleBlob rejected negative sizes, but needle.ReadNeedleBlob
still sized its buffer from the size and is called directly by vacuum and
other paths. Reject a deletion marker before make() there too, and use
size.IsDeleted() in the volume-level checks.
* fix(volume): mirror the blob length check in the rust volume server
write_needle_blob_and_index checked the size against the blob header but
appended the blob verbatim, so a blob that is not the length its size
implies still leaves .dat off the record grid. Match the Go check.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(volume): return an error instead of 201 when a write lands on no volume
ReplicatedWrite only writes locally when this server holds the volume.
For a volume id no server holds, the master lookup returns no locations,
so the write went nowhere and the upload still got 201 Created. The same
happened for a type=replicate write to a server without the volume, so
the primary, or the S3 chunk fan-out, counted a replica that was never
written.
A server without the volume still forwards the write to the replicas the
master lists. When there is nothing to forward to, fail with "volume N
not found on host:port". PostHandler returns that as 500, the status the
Rust volume server already returns here, and uploaders re-assign on 5xx.
Fixes#6609
* volume: reuse Store.HasVolume, drop issue ref from test comment
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix 11388
* fix(volume): scrub validates local deletion tombstones
TombstoneFileSize (-1) is an .idx-only sentinel; the physical record it
points at carries a zero-sized body. Normalize deleted index sizes to 0
via onDiskSize before computing disk usage and calling ReadData, so
corrupted or truncated tombstone records are detected instead of
skipped. Offset-zero entries (remote logical deletes, no .dat record)
remain skipped, and the physical needle id is checked against the index
key. Mirror the behavior in the Rust volume server.
* fix(volume): scrub preserves physical size of deleted non-tombstone entries
Size.Raw()/raw() already encodes the index-to-disk mapping: tombstone
(-1) -> 0, other negative sizes -> their absolute value (the offset then
points at the original record, per the ReadDeleted path). Use it instead
of mapping every deleted size to 0.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(volume): derive needle body tail bound from the version layout
The size guard in ReadNeedleBodyBytes computed the tail length as
checksum, plus timestamp only for Version3. Forks and future on-disk
formats whose tail carries more fields would silently under-check and
still panic in readNeedleTail on a truncated body. Derive the tail from
NeedleBodyLength minus data and padding so the bound stays exact for
every version.
Iterate IsSupportedVersion in the new tests instead of hardcoding
v1-v3 so downstream formats get covered automatically, and skip
versions the build cannot write rather than failing on them.
* test: skip needle write only on the unsupported-version error
A blanket skip would hide a real writer regression. Skip the version
subtest only when the writer reports the version is not supported in
this build (the error text differs between builds), and fail on any
other write error.
* s3api: add Snowflake s3compat API integration tests
Run the upstream snowflakedb/snowflake-s3compat-api-test-suite against a
local SeaweedFS server in CI. test/s3/snowflake/run.sh starts weed server
with S3 (-s3.autoCreateBucket=false so missing-bucket PUTs return
NoSuchBucket), prepares the fixtures the suite needs (versioned bucket,
deny-all-policy bucket, >1000-object prefix), clones the suite, patches
it to path-style addressing, and runs mvn -Dtest=S3CompatApiTest.
The suite also exposed that GetBucketLocation returned 404 NoSuchBucket
for a malformed bucket name; validate the name first and return
400 InvalidBucketName like AWS.
* test: harden snowflake s3compat runner per review
- Pin the upstream suite to a tested commit (SUITE_REV) instead of the
moving default branch
- Bind the test server to loopback only
- Require the AccessDenied error code when verifying the denied bucket
- Fix README so go install runs in a subshell
- checkout with persist-credentials: false
- Make the concurrency group unique per PR, and widen path filters to
the storage/operation/wdclient/cluster/pb packages the S3 stack uses
* test: advertise loopback ip for snowflake test server
-ip.bind 127.0.0.1 alone left the volume server advertising the host's
primary address, so chunk uploads were refused. Also set -ip 127.0.0.1
and disable the Iceberg/Lance listeners so the harness is loopback-only
and does not collide with other local services.
ReadNeedleBodyBytes sliced the needle body with the size from the needle
header without checking it. A corrupted .dat header carrying size -1 still
gets a positive body length (16 bytes on v3), so vacuum compaction read
that body and panicked with "slice bounds out of range [:-1]".
Writers never put a negative size in a .dat header: a delete appends a
size-0 record, and TombstoneFileSize only lives in the .idx. Reject a size
that is negative or leaves no room for the checksum/timestamp tail with an
error wrapping ErrorCorrupted. ScanVolumeFileFrom already logs body read
errors and moves on, so compaction now skips the record like any other
corrupt needle.
Fixes#6763
* pb: stop exiting the process on malformed server addresses
ServerToGrpcAddress and GrpcAddressToServerAddress called glog.Fatalf
when hostAndPort could not parse the port, which os.Exit(255)ed the whole
process. A caller-supplied copy or tail source address reached this path
synchronously in the serving goroutine, so one anonymous VolumeCopy with
a non-numeric port terminated the volume server.
Log the parse error and return the input unchanged instead: the dial or
request that consumes the address then fails as an ordinary error.
* volume: validate copy and tail source addresses before dialing
VolumeCopy, VolumeEcShardsCopy and VolumeTailReceiver dial a
caller-supplied source address (SourceDataNode / SourceVolumeServer)
with no endpoint validation, so an anonymous caller could aim the volume
server at loopback, link-local (cloud metadata) or other unintended
destinations and read dial behavior back as a connectivity oracle.
Apply the same peer-target deny list FetchAndWriteNeedle uses for
replica targets: the source must be a bare host:port whose host is not
loopback, link-local or unspecified; cluster peers stay reachable on
private networks, and -volume.allowUntrustedRemoteEndpoints opts out.
The loopback-using copy tests set the flag to keep exercising the copy
path in process.
* rust volume: validate copy and tail source addresses before dialing
Mirror the Go guard on the Rust volume server: volume_copy,
volume_ec_shards_copy and volume_tail_receiver dial a caller-supplied
source address, so run it through validate_replica_target first (bare
host:port; no loopback, link-local or unspecified hosts; private peers
stay allowed). --volume.allowUntrustedRemoteEndpoints opts out; the test
fixture and the Rust test-cluster launcher set it so loopback sources in
tests keep working.
* volume: pin validated copy/tail source addresses at dial time
validateReplicaTarget resolves the source hostname once, but the gRPC
client resolved it again at connect, leaving a DNS-rebinding window for
hostname sources. The copy and tail source dials now run through the
same guardedDialerPolicy the remote-storage path uses, so every resolved
address is re-checked against the replica deny list (private peers
allowed) immediately before the TCP connect. guardedDialerPolicy also
moves to util.OutboundDialContext so the guarded path keeps the -ip.bind
source binding the default gRPC dialer had.
The Rust volume server mirrors this with connect_guarded, a tonic
connector that resolves, re-checks each address, and connects to the
first passing IP; handlers use it whenever the untrusted-endpoint
opt-out is off. A handler-level test now exercises the enabled
validation branches for all three source-taking RPCs.
* pb: return empty server address for malformed grpc addresses
GrpcAddressToServerAddress used to return the unparseable input on a
hostAndPort failure, so a malformed raft address (e.g. "host:abc")
flowed into admin dashboard master maps unchanged. Return an empty
string instead, skip empty conversions at the two raft-cluster merge
sites, and drop the now-stale comment about the fatal exit the earlier
commit removed.
* test: opt erasure-coding loopback clusters out of the remote endpoint guard
The erasure-coding suites drive VolumeEcShardsCopy / VolumeCopy between
volume servers bound to 127.0.0.1, which the copy/tail source guard now
rejects by default. Pass -volume.allowUntrustedRemoteEndpoints to the
test volume launches, matching what the volume_server framework
harnesses already do.
* admin: only claim fallback master leadership on an empty raft response
A nonempty RaftListClusterServers response whose entries were all
rejected left masterMap empty, so the fallback marked the reachable
current master as leader the same way a genuinely empty (non-raft)
response does. Track whether the successful response returned zero
servers and only promote the fallback master then.
* s3api: test for chunks deleted under an entry the filer committed
Issue #11387: the filer can report a create failure after inserting the
entry (e.g. a parent-directory creation failing post-insert). The error
arrives in the response rather than as a transport status, so it maps
to a definitive error and putToFiler deletes the chunks of the live
entry.
* s3api: confirmCreateLanded also reports a confirmed-absent entry
The verification a failed create runs can answer both directions: the
entry matching the uploaded chunks proves the write landed, and an
authoritative not-found proves the uploaded chunks are orphaned. Return
both outcomes so the cleanup path can gate on the fact rather than the
error class. An empty upload can never prove a landing, so a zero-chunk
entry match no longer upgrades the outcome.
* s3api: delete orphaned chunks only when the entry is confirmed absent
A failed create no longer skips verification based on the error class: the filer can fail after inserting the entry (issue #11387) and a partially-applied routed transaction can leave it behind too, both surfacing as definitive errors. Every failed create now resolves the entry's fate, and the uploaded chunks are deleted only when the entry is confirmed absent; anything unverifiable keeps them for vacuum.
* s3api: confirm absence on every filer the create could have committed on
A lock-path create fails over across filers, so the entry can live on a replica the routed owner has not caught up to; one not-found does not prove absence. The confirmation now queries the owner, the prior owner, and the failover set, declaring absent only when none of them has the entry.
* s3api: bound the reconciliation lookups confirmCreateLanded runs
The lookups ran on context.Background() under the object write lock, so a connected filer that never replies could stall the write path. One timeout now covers the whole enumeration; an expired budget fails the remaining lookups as uncertain, which keeps the chunks.
* fix(filer): leave reader cache unbounded without an explicit budget
NewReaderCache silently installed a 256MiB ReaderCacheBudget when the
caller passed none. Only weed mount opts into a budget; every other
caller (S3 gateway, WebDAV, query engine, mq logstore) inherited the
cap. Under ~90 concurrent S3 GETs of medium objects, prefetch wants
far more than 64 chunk buffers, so reserve() serialized chunk fetches,
clients timed out and retried, and the retry re-downloaded chunks the
cancelled request had already fetched.
A nil budget now means unbounded, restoring the pre-4.47 behavior for
callers that never asked for a memory cap; reserve/complete/release
are nil-safe. The mount path is unchanged and still enforces
-readerCacheSizeMB.
Fixes#11380
* feat(s3): expose -s3.readerCacheSizeMB reader buffer budget
Operators who want the S3 gateway read path memory-bounded can now
opt in: -s3.readerCacheSizeMB on weed filer/server/mini and
-readerCacheSizeMB on standalone weed s3, matching the mount flag.
The default 0 keeps the unbounded pre-4.47 behavior; a positive value
installs a shared ReaderCacheBudget across in-flight and retained
chunk buffers for all S3 GETs.
* fix(filer): validate chunk size before consulting the reader budget
A nil budget returned early and skipped the negative chunkSize check,
letting a corrupted size reach mem.Allocate and panic. Also drop the
command-specific flag prefix from the S3 validation error since
standalone weed s3 exposes the option as -readerCacheSizeMB.
* filer: drop chunk buffers once fully consumed
ReaderCache retained every completed chunk buffer in the downloaders
map until the slot limit evicted it, so buffers lingered after all
readers finished with them.
Track attached readers on each SingleChunkCacher and remove the cacher
when the last reader consumes the buffer to its end. In-flight download
deduplication and the prefetch handoff are unchanged: a buffer always
survives until fully read, partial reads keep it available, and an
attached reader pins a consumed buffer until it detaches. Repeat reads
now go through the chunk cache where enabled, or refetch.
* filer: drop consumed buffers on last detach, rechecked under cache lock
Two review findings on the drop-on-consume change:
- Removal only fired when the detaching reader itself reached the chunk
end. If the end-reaching reader finished first and the last remaining
reader did a partial read or cancelled, the consumed buffer and its
budget reservation lingered until eviction. Track a persistent
consumed flag instead, so any end-reaching read marks the buffer and
the last detach drops it.
- remove() checked only map identity, so a reader attaching between the
reader count hitting zero and removal could attach to a cacher that
was then deleted underneath it. removeConsumed() re-checks identity,
readers == 0, and consumed under the ReaderCache lock; a raced attach
keeps the cacher and its own detach retries the removal.
* s3: abort a completed upload's leftover directory metadata-only
A .uploads/<id> directory can outlive the object it completed into when
the commit's metadata-only removal failed or the gateway died in between;
the restored part entries then share chunks with the published object.
AbortMultipartUpload deleted the directory recursively, chunks and all,
so aborting such a leftover destroyed a committed object (#11382).
Run the same check s3.clean.uploads gained in #11375 before deleting:
when the object entry or a version file under <key>.versions carries the
upload id, remove .uploads/<id> metadata-only and answer the abort; when
the lookup cannot decide, refuse with InternalError rather than risk
live chunks.
* s3: apply the completed-upload check to lifecycle MPU abort
lifecycleAbortMPU ran the same destructive recursive delete on
.uploads/<id>. Reuse uploadCompleted so a leftover whose object entry or
version file carries the upload id is removed metadata-only, and an
undecidable lookup retries later instead of freeing live chunks.
* s3: serialize abort's upload-dir delete with the object's commit
The completed check alone leaves a race: abort can read completed=false,
then an in-flight completion publishes the object over the same part
chunks before the recursive delete frees them.
Run the check and delete inside the object write lock, which non-routed
completions hold for their whole finalize. With an owner, send the data
delete as an ObjectTransaction on the object's lock key — a routed
commit then either loses its upload-exists precondition after our delete
or has already stamped the object, which the transaction's
IF_EXTENDED_NOT_EQUAL condition detects and falls back to a
metadata-only remove. lifecycleAbortMPU shares removeUploadDir so both
callers get the same ordering.
* s3: check for an empty object before resolving its write owner
* s3: check completion at the abort's resolved object key
An upload record missing ExtMultipartObjectKey skipped the completed
check entirely even though the request's Key names the object.
* s3api: fail closed when S3 Tables signature verification fails
* s3api: avoid nil Account dereference in S3 Tables auth log
* iceberg: return auth error instead of falling back to DefaultAllow
* lance: return auth error instead of falling back to DefaultAllow
* s3api: stop trusting client-supplied s3-account-id
The header is set by the server after successful authentication; scrub
inbound values alongside the other internal headers, and apply the same
admin guard to the header fallback branch of getAccountID that the
identity branch already has.
* test: cover table-catalog auth wrappers and principal resolution
* test: configure anonymous identity where catalog clients do not sign
* s3api: scrub s3-account-id after signature verification
* s3api: test that a session token must not reveal its credential
* sts: derive secret access key with HMAC keyed on the signing key
* s3api: stop accepting STS session tokens as bearer credentials
* security: reject STS session tokens on filer and admin gRPC auth
* test: sign s3/iam framework requests with the session credential
* s3api: exercise the real auth pipeline in the end-to-end harness
* s3: commit versioned multipart upload in one transaction
CompleteMultipartUpload wrote the version file, flipped the .versions
pointer, then removed .uploads/<id> metadata-only as a best-effort
post-commit step. A filer error or gateway crash in that window left the
upload directory referencing the same chunks as the published object, and
the next s3.clean.uploads run purged it with data -- corrupting a
committed object.
Put the version file, remove the upload directory metadata-only (its
chunks are the object's chunks), and recompute the latest pointer in one
ObjectTransaction under the object's per-path lock on the owner filer.
The mutation order keeps every partial state safe: the chunks stay
referenced at all times, and a published object never coexists with the
upload directory the cleaner would purge.
Unused part entries are freed before the transaction, since the
metadata-only directory delete would otherwise leak their chunks.
* s3: remove upload directory inside the multipart object PUT
The same committed-object/stranded-upload window existed on the
suspended and non-versioned paths: writeMultipartObject committed the
object, then a best-effort rm dropped .uploads/<id>. Ride the
metadata-only removal on the routed PUT itself so the two land in one
transaction; the unrouted mkFile fallback keeps post-commit cleanup.
* shell: purge completed uploads metadata-only in s3.clean.uploads
A leftover .uploads/<id> can outlive a committed object when the
completion's metadata-only delete fails or the gateway dies in between;
its part entries then share chunks with the live object, and a recursive
purge frees them out from under it.
Before purging a stale upload, check whether it completed: the object
entry or any version file under <key>.versions carrying the upload id.
If so, delete with skipChunkDeletion. If the lookup fails, skip the
upload for this run rather than risk live chunks.
* s3: abort multipart completion when unused part cleanup fails
Deleting the upload directory metadata-only erases the only metadata
pointing at part entries whose deletion failed, orphaning their chunks.
Propagate the error so the completion fails while the upload directory
still exists and the request remains retriable.
* s3: require the upload directory to exist at multipart commit
A delete that does not take the object lock (abort, lifecycle,
s3.clean.uploads) can remove .uploads/<id> and its chunks between the
prepare step and the commit transaction. The commit now carries an
IF_EXISTS precondition on the upload directory so the race fails the
request with NoSuchUpload instead of publishing an object over freed
chunks.
* s3: keep the version file when the upload directory is gone
The finalize transaction has no rollback, so a failure at the
latest-pointer recompute leaves the version written and .uploads/<id>
removed. Deleting the version then destroys the only remaining record of
the upload, making a retried CompleteMultipartUpload return NoSuchUpload
while the version's chunks leak. Roll back only while the upload
directory survives; otherwise keep the version, which a retry resolves
through SeaweedFSUploadId and the version reconciler promotes.
* s3: keep manifests when a routed object write partially commits
For non-versioned and suspended completions the object PUT precedes the
upload-directory DELETE, so an error can mean the object entry exists
while the response reports failure. Freeing this attempt's manifest
chunks then destroys the committed object. Keep them when the object
entry survived, and after a failed null-marker finalize which always
follows a committed write.
* s3: skip the keep-version path on precondition failure
A rejected precondition means no mutation ran, so there is no version
file to preserve and this attempt's manifests are orphans the error
cleanup should free.
* s3: keep manifests when the object-existence check itself fails
A transient lookup error previously read as absent, letting the error
cleanup free manifest chunks a committed object still references.
* s3: keep the upload directory when post-commit part cleanup fails
Removing it metadata-only after a failed entry delete erases the only
reference to the leftover chunks. Leave the directory so the entries
keep their chunk references for s3.clean.uploads or manual recovery.
* pb: fix filer list entry counting on 32-bit
int(limit) wraps to -1 on 386 when limit is math.MaxUint32, so the
beyond-limit check discarded every streamed entry. Compare in uint64
instead; the semantics are unchanged on 64-bit platforms.
* shell: resolve trailing-slash object keys in s3.clean.uploads
Completion stores a key ending in / inside the directory it names
(<bucket>/dir/dir), but FullPath+DirAndName on the normalized key
looked one level too high. Deriving dir and name with path.Dir and
path.Base mirrors getEntryNameAndDir so the completed-upload check
finds the entry instead of purging its chunks.
* s3: heal a suspended completion hidden behind a delete marker
Removing .uploads/<id> inside the commit transaction means a failed
finalizeSuspendedNullWrite leaves nothing to retry against: the object
entry is committed but the marker still makes the key read as deleted,
and a retried CompleteMultipartUpload can only report NoSuchUpload.
When the upload directory is gone, check the regular path for an entry
carrying the upload id and re-run the marker finalize, so the retry
both succeeds and repairs the key. Only suspended buckets can hold
this state; anything newer owns the key.
* s3: report store errors when resuming a committed multipart object
* mount: keep a deliberate bucket allow-empty-folders setting
* shell: s3.bucket.allowEmptyFolders toggles the empty folder cleaner
* shell: guard allow-empty-folders toggle with expected extended attrs
* filer: drop cached empty-folder policy on bucket entry update
* mount: guard allow-empty-folders write with expected extended attrs
* filer: skip caching a stale cleanup policy read across an update
* mount, shell: snapshot the full extended attributes for update preconditions
* filer: fail closed and invalidate on all bucket entry events for cleanup policy
* filer: key the cleanup policy generation by bucket
* filer: requeue cleanup when the bucket policy cannot be loaded
* filer: expire idle cleanup policy generations
* filer: skip requeueing cleanup after the cleaner stops
* filer: bound cleanup retries on repeated policy failures
* filer: cover cleanup requeue on repeated policy failures
* filer: keep cleanup policy generations while reads are in flight
* filer: exercise the cleanup queue lifecycle in the retry-cap test
* telemetry/server: tidy module dependencies
* ci: make telemetry deploy work on Oracle Linux 7
* ci: install telemetry unit and logrotate on every deploy
* ci: abort telemetry deploy on install failures
* s3api: map ambiguous filer transport errors to retryable 503
Canceled, DeadlineExceeded and Unavailable can be returned after the
filer applied the write, so the outcome is ambiguous. Reporting them as
a 4xx tells the client not to retry; report ServiceUnavailable instead.
* s3api: verify entry existence before deleting orphaned chunks
A failed CreateEntry can still have landed on the filer when the error
is a transport failure, and entryCreated=false would tombstone chunks a
live entry references, leaving a dangling pointer that survives only
because reads pass readDeleted=true until vacuum reclaims the needle.
Before deleting, look the entry up: if it is stored with the same
chunks, the write succeeded; if the lookup cannot be answered, keep the
chunks for vacuum to reclaim; only a confirmed absence still cleans up.
* s3api: regression tests for ambiguous CreateEntry outcomes
Covers the three post-create-failure cases in putToFiler: the entry
landed despite the error (treat as success, keep chunks), the entry is
confirmed absent (delete orphans), and the outcome is unverifiable
(keep chunks, return error).
* volume: count reads served from deleted needles
A readDeleted read succeeding on a tombstoned needle is the signal that
metadata still points at deleted data. Count it under a
readDeletedNeedle handler label in both the Go and Rust volume servers
so the condition is visible before vacuum turns it into a 404.
* s3api: never delete chunks on an ambiguous create error
Review feedback on the first fix showed verification could still go
wrong in both directions: a stale or lagged lookup could report
not-found for a committed entry, a prefix object stores its chunks on a
directory entry, and filer-side manifestization rewrites the top-level
chunk ids the comparison relied on.
Rework the rule so the outcome classes are asymmetric:
- A transport-level error (anything filerErrorToS3Error maps to a
retryable 503) is ambiguous and never deletes chunks; the lookup can
only upgrade the write to success.
- Any other error is a definitive filer refusal and still cleans up.
confirmCreateLanded asks the write owner first, resolves the stored
entry through chunk manifests, requires an exact match of the uploaded
file ids, and on success runs the finalize callback the failed create
skipped (under the object write lock, with the same rmObject undo the
create path uses). Zero-chunk writes stay ambiguous since they cannot
be told apart by chunks.
* s3api: cover definitive refusals and stale entries in put tests
The confirmed-failure case now uses a definitive refusal so it still
exercises orphan cleanup, and a new case keeps chunks when the stored
entry belongs to an older object rather than this PUT.
* volume: count deleted-needle reads once per request
Streamed Go reads ran the deleted check in readNeedle and again in
readNeedleDataInto, and non-streamed Rust reads in stream_info and the
full-read fallback, double-counting one request. Count at the single
entry probe each implementation takes per GET: readNeedle in Go,
read_needle_stream_info in Rust.
* s3api: run recovered-write rollback under the object lock
Two follow-ups from review: ResolveChunkManifest returns traversed
manifest blobs in its manifestChunks output, so requiring it empty
rejected every manifestized landing; and the rmObject undo ran after
the object write lock was released, so a concurrent newer write could
be deleted between finalize failure and rollback. Compare only the
resolved data chunks and keep the undo inside the lock.
* s3api: verify, finalize and roll back recovered creates in one lock
A lookup done before the object write lock let a concurrent PUT replace
the entry between the chunk comparison and the finalize/rollback
section, so a failed afterCreate could rmObject a newer write. Run the
owner lookup, manifest resolution, chunk comparison, afterCreate and
the conditional undo inside a single withObjectWriteLock section.
* security: require go 1.26.6 and bump vulnerable deps
A dependency scan of the 4.47 release flagged the bundled toolchain
and modules:
- github.com/golang/go < 1.26.6 (CVE-2026-39821, CVE-2026-56853,
CVE-2026-56859, CVE-2026-56862, CVE-2026-56864, CVE-2026-56865,
CVE-2026-33818, CVE-2026-46600): raise the go directive to 1.26.6
so every built artifact requires the fixed toolchain.
- google.golang.org/grpc (CVE-2026-84445, CVE-2026-84304): move to
the fixed dev pseudo-version; released tags through v1.85.0-dev
remain in the affected range.
- github.com/pelletier/go-toml/v2 <= v2.4.2 (unbounded parser
recursion): v2.4.3.
- alpine libcrypto3/libssl3 < 3.5.8-r0 (CVE-2026-75803,
CVE-2026-63073, CVE-2026-63075, CVE-2026-63076, CVE-2026-63072,
CVE-2026-54874, CVE-2026-18798, CVE-2026-14456, CVE-2026-14457):
the release images already apk-upgrade the final stage; extend the
same to the telemetry and admin-integration images.
Same bumps applied to the test/kafka, test/sftp,
kafka-client-loadtest, and telemetry/server modules.
* telemetry: send integration test report above the 10 GiB floor
The collect endpoint keeps reports only when TotalDiskBytes >=
proto.MinDiskBytes, but the integration test still sent 1 GiB, so the
server counted the report and skipped storing it. No cluster_id series
was ever created and /metrics lacked seaweedfs_telemetry_volume_servers.
Send just above the floor (via proto.MinDiskBytes so it cannot silently
drift again) so the expected per-cluster metrics are exported.
* rust volume: one positional read helper; never seek a dup'd handle on Windows
Positional read-exact was hand-rolled four times: the complete
cross-platform version in needle_map/sorted_file.rs, a Windows-only half
in volume.rs whose unix half was inlined as a
cfg(unix)/cfg(windows)/compile_error! triple at three call sites, a
byte-identical Windows-only copy in ec_volume.rs, and read_full_at in
ec_bitrot.rs. Three more sites -- EcVolumeShard::read_at,
EcLocalShard::read_at and ec_encoder::read_at_most -- hand-rolled the
short-read-permitted variant with a cfg(not(unix)) arm that
try_clone()s the handle and seeks it.
That last arm is wrong. A duplicated descriptor shares one kernel file
offset with the original, so seek-then-read is two syscalls against
state another thread can move in between: a concurrent reader or an
append repositions the offset and the read returns bytes from somewhere
else entirely. EcLocalShard::read_at documents that it must never seek,
one line above the seek. Windows seek_read carries its own offset in a
single call, so that window does not exist.
All seven now go through storage::io::{read_exact_at, read_at}, whose
module doc records why duplicating a handle is not a way to get a
private file position -- opening the file again is, as
Volume::dat_scan_plan already does. read_at_most keeps its own
fill-until-EOF loop; only the per-iteration positional read changes.
Behaviour on unix is unchanged: every unix arm was already
FileExt::read_exact_at or FileExt::read_at. The one exception is
ec_bitrot::verify_shard_blocks, which now retries on EINTR (std's
read_exact_at does; the loop it replaces did not) and, on unix, reports
the standard "failed to fill whole buffer" text instead of "short read
on shard block". The Windows arm still says "unexpected EOF in
seek_read"; both carry ErrorKind::UnexpectedEof, as before.
NeedleStreamSource::read_exact_at and Volume::read_exact_at_backend keep
their signatures; only their bodies shrink.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: retry Interrupted in Windows read_exact_at
Unix std's FileExt::read_exact_at ignores ErrorKind::Interrupted and
retries, but the Windows seek_read loop propagated it, so the shared
exact-read contract differed by platform. seek_read can surface
ERROR_OPERATION_ABORTED, which std maps to Interrupted.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* s3tables: hide inaccessible table buckets
* s3tables: hide inaccessible namespaces
* s3tables: hide inaccessible tables
* s3tables: hide inaccessible resources in rename and namespace delete
RenameTable/RenameView denied on the source now report the same
not-found as a missing source, and the destination name conflict is
checked only after destination authorization so a denied caller cannot
distinguish an existing destination namespace or name from a missing
one. DeleteNamespace denials use the same formatted message as a
missing namespace.
* fix(ec): BatchDelete cookie fail-closed via locate_data geometry
* fix(ec): honor skip_cookie_check, require full cookie header
* fix(ec): retry short cookie header reads, still fail closed on EOF
* chore(ec): trim cookie validation comments
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Serve can return when its listener closes while HTTP requests are still draining. The main path could then close the metadata store before those requests finish.
Make signal, context, and Serve-exit paths join one shutdown sequence. Drain gRPC and HTTP concurrently with 15-second default limits, then close the store. Test both completion orders.
fix: flush metadata log before closing filer store
Serialize sealed-batch handoffs with shutdown, reject late appends, and wait for log-buffer workers before closing the filer metadata store.
Cover queued writes, interval and explicit flushes, late-write rejection, and pending persistence with shutdown tests.
handlers.rs split needle URLs in three places and the three disagreed.
Go does it once, in parseURLPath (weed/server/common.go:218-249), and
dispatches on the slash count: /vid/fid/filename takes the extension off
the filename and leaves the fid whole, /vid/fid takes it off the fid, and
the comma form splits the last segment on its last comma and dot.
Two of the Rust copies got that wrong:
- extract_file_id returned the path unchanged when it found no comma, so
a JWT fid claim, which Go compares against vid + "," + fid for every URL
form (volume_server_handlers.go:361-364), could never match a slash-form
request. With a JWT key configured, every read, write or delete of
/3/01637037d6 was a 401.
- build_proxy_request_info's slash branch had no extension handling, so a
redirect for /3/01637037d6.jpg sent the client to /3,01637037d6.jpg.
Go's proxyReqToTargetServer formats "%s/%s,%s" from the already-stripped
fid (volume_server_handlers_read.go:128-137) and so emits
/3,01637037d6. The peer still serves either form, since the comma form
strips the extension again, so this one is parity rather than breakage.
Replace all three with one parse_needle_path returning vid, fid, ext and
filename borrowed from the path. The fid keeps its _delta suffix, as in
Go: parse_needle_id_cookie applies it and the JWT check strips it. The
leading slash stays optional, so chunk manifest fids still parse.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* shell: fs.verify -pruneEntries deletes entries whose needles are lost
* shell: harden fs.verify -pruneEntries guards; VolumeNeedleStatus returns NotFound for absent needles
* shell: resolve chunk manifests in fs.verify metadata path; require confirmed deletion before counting prunes
* shell: anchor fs.verify legacy missing-needle error matching
* shell: keep fs.verify metadata scan alive on manifest resolution failures
* shell: classify EC missing needles and keep manifest failures unverified
VolumeNeedleStatus now canonicalizes erasure_coding.NotFoundError to
codes.NotFound, so absent needles in EC volumes reach the prune path
through the same stable contract as regular volumes. The client-side
isNeedleMissingError keeps recognizing the legacy wrapped EC shape
("locate in local ec volume: ... needle not found") for mixed-version
clusters.
A chunk manifest that fails to resolve is now an entry-level
verification failure even when the raw top-level chunks are healthy:
the file is not fully readable without the manifest. Raw chunks are
still verified on a resolution failure so a missing top-level manifest
needle is classified and can be pruned. The per-entry logic is
extracted into resolveAndVerify for testability.
* shell: trim fs.verify prune comments
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: a list marker that sorts before the prefix excludes nothing
ListObjects `marker` and ListObjectsV2 `start-after` are a plain key
cutoff: list the keys that sort after it. A marker that sorts before the
prefix and is not under it therefore excludes no key carrying the prefix,
and the listing must equal the one with no marker at all.
normalizePrefixMarker treated every marker that does not start with the
prefix as "something wrong" and the listing came back empty. Clients send
this shape routinely: docker/distribution's S3 storage driver walks
prefix "<root>/<path>/" with start-after "<root>" (its rootdirectory), so
on SeaweedFS a registry walk saw an empty bucket. zot read that as "no
repositories": /v2/_catalog was empty, GC/scrub/retention never saw a
repo, and on restart its storage parse deleted every repository's
metadata as "no longer in storage".
listFilerEntries now lists as if no marker were given when the marker
sorts before the prefix; the response still echoes the marker the client
sent. A marker that sorts after the prefix's subtree is left alone: it may
legitimately sit inside a partial-name prefix's match set, which
normalizePrefixMarker already handles, and otherwise correctly lists
nothing.
Reproduce on 4.44 and 4.47:
curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=zot/zot/" # all keys
curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=zot" # KeyCount 0
curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=a" # KeyCount 0
* s3: keep the prefix's own key excluded by a marker that names it
Fold the before-prefix marker rule into normalizePrefixMarker, which now also
derives prefixEndsOnDelimiter from the effective marker instead of each cursor
rebuilding the expression.
A marker equal to the prefix is no longer trimmed to a subtree cutoff:
start-after "a/b/" with prefix "a/b/" excludes only the "a/b/" key, so the walk
starts inside that directory and its children still list.
Adds a listing-level test that walks the whole path for both start-after shapes
a registry sends, and covers the new normalization cases.
* s3: leading slashes do not hide a marker that names the prefix
* s3: echo the V1 marker the client sent, not the walk's cutoff
* s3: filter only the walk's cutoff from the V1 page, not the echoed marker
* s3: skip the key an exclusive marker names as it streams
---------
Co-authored-by: Zuse <be9c90a8-c104-4be2-b7a4-9f92eb833ac8@forge.local>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust volume: add a .dat scan plan that runs without the store lock
DatScanPlan captures a fresh .dat handle, the version, the start offset
and an end bound while the caller holds a store guard, then visits one
record at a time with positional reads that never touch the Volume, the
way Go's ScanVolumeFileFrom feeds a scanner.
The handle pins the inode the offset was resolved against: a vacuum
commit renames .cpd over .dat and destroy unlinks it, and neither
rewrites the pinned bytes. The end bound is read while no writer can
hold store.write(), so the scan never meets a partial append. It is a
fresh open, not try_clone, because on Windows read_exact_at uses
seek_read, which moves a cursor a clone shares with the writer.
A header whose size is negative, or does not fit before the end bound,
ends the pass before the body length is computed or anything is
allocated. In today's scan a negative size reaches needle_body_length
and either overflows the buffer size or walks the scan from a wrong
offset. A size near i32::MAX overflows padding_length's i32 arithmetic,
which panics in debug builds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3
* rust volume: stream the tail scan with the store lock released
volume_tail_sender read every needle from the start offset to EOF into a
Vec while holding store.read(). volume.merge tails from zero, so that was
the whole volume in memory. And because needle writes and the heartbeat
take store.write() on a lock that prefers writers, the whole node stopped
serving until the scan finished: the failure #11235 fixed for EC scrub.
Each pass now runs on a blocking thread. Under one store guard it
resolves the start offset and captures a DatScanPlan, then drops the
guard and sends each needle as it is read, as Go's
VolumeFileScanner4Tailing does. This replaces the one-guard-across-
search-and-scan rule from the previous commit with a stronger invariant:
the offset, the handle and the end bound come from the same guard, and
the handle pins the inode, so a vacuum commit mid-scan cannot point the
offset into the compacted file.
A scan error now ends the stream with Status::internal instead of a
clean EOF, as Go's `streamFollow: %w` does. Once needles stream, a clean
EOF after a partial pass would let volume.move treat a truncated tail as
complete. A panic in the pass is reported the same way. A receiver that
hangs up is also noticed between skipped needles, not only on a send.
Unchanged: the append_at_ns filter, the header on every 2MB chunk, the
caught-up heartbeat without a scan, and the draining countdown.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3
* rust volume: fail the tail pass on a short read below the snapshot end
DatScanPlan::scan treated an UnexpectedEof on the header or body read as
the end of the data and returned Ok. Every byte below the captured end
existed when the plan was taken, so a short read there can only mean the
inode was truncated under the plan: an unmount followed by a VolumeCopy
of the same volume id reopens .dat with truncate(true). The pass then
reported Scanned, the next pass found the volume gone, and the stream
ended cleanly after a prefix of the planned records, which volume.move
would take as a complete tail.
Both short-read arms now fail the scan with an I/O error that names the
offset and the snapshot end, so tail_pass reports Status::internal as it
does for every other read failure. The break arms were carried over from
scan_raw_needles_from, where the whole scan ran under the store guard and
nothing could truncate the file.
Found by the Devin and Greptile reviews on #11275.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: sum the needle padding in i64 so a corrupt size cannot overflow
padding_length added the header, checksum and timestamp widths to the
needle size in i32. A size read from a corrupt header can sit near
i32::MAX, and that sum then overflows: a panic with overflow checks, a
wrapped padding without. DatScanPlan::scan bounds the size against the
bytes left before computing the body length, but that only keeps such a
size out of the arithmetic while under 2 GiB of the file remains, so on a
large volume the scan could still reach the overflow and, in release,
size a buffer from garbage.
Sum in i64 in both version branches. The result is at most
NEEDLE_PADDING_SIZE, so it still fits Size. The scan comment no longer
claims the bound check prevents the overflow.
Found by the CodeRabbit review on #11275.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: propagate dat scan parse failures
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: roll master pods when master config changes
The master loads master.toml once at startup (startAdminScripts reads
master.maintenance.scripts and sleep_minutes via viper with no config
watching), and the master ConfigMap is mounted with subPath, which
kubelet never refreshes in a running pod. So a change to
.Values.master.config today updates the ConfigMap but running masters
keep executing the old configuration until something else restarts
them.
Add a checksum/config annotation on the master pod template, following
the existing checksum/s3config pattern on the filer and s3 pods, so a
master config change triggers a rolling restart of the masters.
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* Guard against duplicate keys
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* Add checksum to deployment as well
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* Always ensure the annotation is set
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* Update comments
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* Soften stance
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* helm: merge pod annotations before checksums
---------
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust: migrate seaweed-volume and seaweed-worker to tonic 0.14 / prost 0.14
tonic 0.14 boxes the contents of tonic::Status, which is what made every
RPC path trip clippy's result_large_err; the allow for that lint goes in
the next commit. The prost codec moved out of tonic into tonic-prost and
tonic-prost-build, so both build scripts now call
tonic_prost_build::configure() and both crates depend on tonic-prost for
the generated code. The `tls` feature was split into a per-backend
feature; `tls-aws-lc` is the same backend both crates already install
through rustls::crypto::aws_lc_rs.
tonic 0.14 depends on axum 0.8 and tower 0.5, which would have left a
second axum and a second tower in each tree next to the 0.7 / 0.4 the
crates named themselves. Bumping them keeps one copy of each: axum 0.8
only changes the path-parameter syntax for the routes here (`/:vid` ->
`/{vid}`, `/*path` -> `/{*path}`), tower 0.5 needs the `util` feature
named explicitly for ServiceExt::oneshot (it used to arrive through
tonic's feature unification), and tower-http 0.6 is the matching
release.
Lock files move only through cargo's own resolution for the new
versions; no other dependency was refreshed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust: drop the result_large_err allow now that tonic::Status is boxed
tonic 0.14 stores Status behind a Box, so Result<_, Status> is no longer
a large-Err type and clippy has nothing to say about it. Both crates
pass `cargo clippy --all-targets -- -D warnings` without the allow
(seaweed-volume in both feature sets), so the policy entry and its
comment go.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: drop the unused headers argument of try_expand_chunk_manifest
The parameter was already named `_headers`; nothing in the body reads it.
With it gone the function is under clippy's argument threshold and the
expect goes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: pass EC peer reads an EcInterval instead of ten arguments
fetch_one_interval, read_remote_ec_shard_interval,
do_read_remote_ec_shard_interval and recover_one_remote_ec_shard_interval
all took the same (vid, needle_id, shard_id, shard_offset, size,
expected_encode_ts_ns) tuple, and the two that reconstruct also took the
location map with the data/parity counts. Those are now EcInterval (Copy)
and EcShardMap (a borrow of the map plus the counts). The fan-out inside
recovery builds its per-shard request with `EcInterval { shard_id: sid,
..iv }`, which is the one place the old argument list was easy to get
wrong. Bodies destructure at the top, so the code below the signatures
is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: give the EC encoder an EcEncodeLayout and an EncodeRun
encode_dat_file took the Reed-Solomon shape and three block sizes as five
loose integers; they are now one Copy struct, EcEncodeLayout, which is
what Go calls ECContext. The per-row and per-batch helpers took the same
six sinks and the offsets; they become methods on EncodeRun, which owns
the borrows for one run, so each call names only the offset and block
size that vary. The byte-level work is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: describe a .dat rebuild with DatRebuild instead of nine arguments
write_dat_file_from_shards, its _with_dirs twin and the private
write_dat_file were three layers over one nine-argument signature. One
public function now takes a DatRebuild, whose shard_dirs is None when
every shard sits beside the .dat and Some(dirs) for the cross-disk
reconciled layout. The field docs carry what the function doc used to
say about the encode-time size and the block layout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: split copy_file_from_source's fifteen arguments into two structs
CopyFileSpec is the per-file request (what to ask the source for, where
it lands, whether its bytes count as progress); CopyProgress is the
sender, throttler and report state that all three files of one
VolumeCopy share, held by &mut across the calls. The three production
call sites now read as the .dat/.idx/.vif literals they are, instead of
positional trues and falses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: create volumes from a VolumeSpec
Volume::new, DiskLocation::create_volume and Store::add_volume each
took the same five-value tail of Go's NewVolume argument list:
collection, replica placement, TTL, preallocation and needle version.
That tail is now VolumeSpec, a Copy struct whose Default is what almost
every test wanted anyway (empty collection, no replication, no TTL, no
preallocation, current version), so most of the 104 call sites shrink
to `&VolumeSpec::default()` or name the one field they set. The id,
directories, index kind and disk type stay positional because they
differ at every site.
Two imports that only test modules use moved into those modules, and
DiskLocation no longer imports ReplicaPlacement.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-worker: run cargo fmt
Layout only; no token in the workspace changes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: run cargo fmt
Layout only; no token in the crate changes. Every earlier Rust PR here
formatted only the blocks it touched so as not to drown its diff in
this one, and this commit is that debt paid in a single place. rustfmt
needed two passes to settle one block in handlers.rs; the committed
form is the fixed point, so `cargo fmt --check` is clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* ci: add a commented-out cargo fmt --check step to both Rust workflows
Same shape as the commented clippy step from #11312: the check is
written out so that making formatting a gate is a one-line uncomment,
and whether to do that stays a maintainer call.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* rust: migrate seaweed-volume and seaweed-worker to tonic 0.14 / prost 0.14
tonic 0.14 boxes the contents of tonic::Status, which is what made every
RPC path trip clippy's result_large_err; the allow for that lint goes in
the next commit. The prost codec moved out of tonic into tonic-prost and
tonic-prost-build, so both build scripts now call
tonic_prost_build::configure() and both crates depend on tonic-prost for
the generated code. The `tls` feature was split into a per-backend
feature; `tls-aws-lc` is the same backend both crates already install
through rustls::crypto::aws_lc_rs.
tonic 0.14 depends on axum 0.8 and tower 0.5, which would have left a
second axum and a second tower in each tree next to the 0.7 / 0.4 the
crates named themselves. Bumping them keeps one copy of each: axum 0.8
only changes the path-parameter syntax for the routes here (`/:vid` ->
`/{vid}`, `/*path` -> `/{*path}`), tower 0.5 needs the `util` feature
named explicitly for ServiceExt::oneshot (it used to arrive through
tonic's feature unification), and tower-http 0.6 is the matching
release.
Lock files move only through cargo's own resolution for the new
versions; no other dependency was refreshed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust: drop the result_large_err allow now that tonic::Status is boxed
tonic 0.14 stores Status behind a Box, so Result<_, Status> is no longer
a large-Err type and clippy has nothing to say about it. Both crates
pass `cargo clippy --all-targets -- -D warnings` without the allow
(seaweed-volume in both feature sets), so the policy entry and its
comment go.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: drop the unused headers argument of try_expand_chunk_manifest
The parameter was already named `_headers`; nothing in the body reads it.
With it gone the function is under clippy's argument threshold and the
expect goes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: pass EC peer reads an EcInterval instead of ten arguments
fetch_one_interval, read_remote_ec_shard_interval,
do_read_remote_ec_shard_interval and recover_one_remote_ec_shard_interval
all took the same (vid, needle_id, shard_id, shard_offset, size,
expected_encode_ts_ns) tuple, and the two that reconstruct also took the
location map with the data/parity counts. Those are now EcInterval (Copy)
and EcShardMap (a borrow of the map plus the counts). The fan-out inside
recovery builds its per-shard request with `EcInterval { shard_id: sid,
..iv }`, which is the one place the old argument list was easy to get
wrong. Bodies destructure at the top, so the code below the signatures
is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: give the EC encoder an EcEncodeLayout and an EncodeRun
encode_dat_file took the Reed-Solomon shape and three block sizes as five
loose integers; they are now one Copy struct, EcEncodeLayout, which is
what Go calls ECContext. The per-row and per-batch helpers took the same
six sinks and the offsets; they become methods on EncodeRun, which owns
the borrows for one run, so each call names only the offset and block
size that vary. The byte-level work is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: describe a .dat rebuild with DatRebuild instead of nine arguments
write_dat_file_from_shards, its _with_dirs twin and the private
write_dat_file were three layers over one nine-argument signature. One
public function now takes a DatRebuild, whose shard_dirs is None when
every shard sits beside the .dat and Some(dirs) for the cross-disk
reconciled layout. The field docs carry what the function doc used to
say about the encode-time size and the block layout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: split copy_file_from_source's fifteen arguments into two structs
CopyFileSpec is the per-file request (what to ask the source for, where
it lands, whether its bytes count as progress); CopyProgress is the
sender, throttler and report state that all three files of one
VolumeCopy share, held by &mut across the calls. The three production
call sites now read as the .dat/.idx/.vif literals they are, instead of
positional trues and falses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: create volumes from a VolumeSpec
Volume::new, DiskLocation::create_volume and Store::add_volume each
took the same five-value tail of Go's NewVolume argument list:
collection, replica placement, TTL, preallocation and needle version.
That tail is now VolumeSpec, a Copy struct whose Default is what almost
every test wanted anyway (empty collection, no replication, no TTL, no
preallocation, current version), so most of the 104 call sites shrink
to `&VolumeSpec::default()` or name the one field they set. The id,
directories, index kind and disk type stay positional because they
differ at every site.
Two imports that only test modules use moved into those modules, and
DiskLocation no longer imports ReplicaPlacement.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* pb: add multipart concurrency fields to RemoteConf and tier move requests
RemoteConf gains upload_concurrency/download_concurrency (0 = client
default); VolumeTierMoveDatToRemote/FromRemote requests gain a
concurrency field (0 = backend default).
* remote storage: honor RemoteConf upload/download concurrency in s3 and azure clients
s3 client: ReadFile passes conf download_concurrency to the downloader,
WriteFile uses upload_concurrency for the uploader; previously
hard-coded 1 upload / 5 download parts. 0 keeps defaults. Same for
azure client.
* storage: plumb concurrency through backend interface and tier upload/download
BackendStorage.CopyFile/DownloadFile take a concurrency hint (<=0 =
backend configured default); s3 backend reads
upload_concurrency/download_concurrency from scaffold config with
parseConcurrency fallback, rclone updated to the new signature. Tier
move gRPC handlers forward the request concurrency to the backend.
* shell: -upload_concurrency/-download_concurrency for remote.configure, -concurrent for volume.tier
remote.configure exposes upload/download concurrency persisted into
RemoteConf; volume.tier move/evict commands forward -concurrent to the
tier move requests. Documented in master-cloud.toml scaffold.
* test: cover concurrency propagation in remote tier integration test
* remote.configure: merge existing config on partial update
Load the stored RemoteConf before saving so a partial update (e.g. only
-upload_concurrency) preserves credentials, endpoints, and type instead
of replacing them with new-config defaults. Only treat a confirmed
ErrNotFound as a new configuration; propagate all other load errors so a
transient filer failure does not overwrite stored settings.
On a type transition, reset backend-specific fields to the destination
type's new-config defaults rather than inheriting the old backend's
empty values. Bound configured concurrency to a sane maximum.
* remote storage: honor configured download concurrency in S3 and Azure
ReadFileWithConcurrency now resolves a zero request override against the
client's configured download_concurrency (new downloadConcurrency()
helpers), so the remote-mount/cache read path honors
RemoteConf.DownloadConcurrency instead of the hard-coded default.
Azure also clamps the resolved value to math.MaxUint16 regardless of
whether the fallback was used, preventing uint16 wraparound when a
configured value exceeds 65535.
* shell: rename -concurrent to -concurrency and validate tier transfer bounds
Rename the -concurrent flag to -concurrency across volume.tier.upload,
volume.tier.download, and volume.tier.compact to match the proto field and
RemoteConf field names. Add validateTierConcurrency to reject values that
would wrap int32 or exceed a 1024 cap before constructing the request.
* server: clamp tier move concurrency in gRPC handlers
Add clampTierConcurrency to both VolumeTierMoveDatToRemote and
VolumeTierMoveDatFromRemote handlers so a direct gRPC caller cannot spawn
an unbounded number of network workers.
* trim verbose comments added with concurrency feature
Remove redundant doc comments on the backend interface, rclone backend,
s3_backend parseConcurrency, and test helpers that restated the obvious.
* remote.configure: apply type defaults before re-parse so explicit flags win
applyTypeDefaults ran after the second flag parse, overwriting explicit
destination flags (e.g. -s3.region=eu-west-1) with new-config defaults.
Move the type-transition default reset before the re-parse so user-supplied
flags override the destination defaults.
* remote.configure: only treat explicit -type as a type transition
The first parse defaults -type to s3, so a concurrency-only update on an
existing non-S3 config captured requestedType=s3 and wrongly triggered a
type transition, resetting the stored backend to S3. Use fs.Visit to
detect whether -type was explicitly supplied; an omitted -type keeps the
stored backend.
---------
Co-authored-by: Jack Meredith <9480542+jackusm@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: preserve versions for POST policy uploads
Route POST policy uploads through the existing version-aware write helpers
and validate promoted Object Lock headers before writing.
Return the generated version ID when versioning is enabled, return
x-amz-version-id: null when versioning is suspended, and omit the header
when versioning has never been enabled.
* s3: reuse versioning helpers in POST policy handler
Route the POST policy handler through the existing getVersioningState
and isObjectLockEnabled helpers instead of open-coding the object-lock
forces-versioning-enabled rule, matching the PUT path.
Drop the x-amz-version-id: null response header for suspended
versioning; the PUT handler omits it and the S3 PutObject sample
response for suspended buckets does not include it. Trim the moved
fileSize comment.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* build(deps): bump github.com/redis/go-redis/v9 from 9.21.0 to 9.22.0
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.21.0 to 9.22.0.
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.21.0...v9.22.0)
---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
dependency-version: 9.22.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* test(redis_conf): track go-redis 9.22.0 default read timeout of 5s
go-redis 9.22.0 raised the default ReadTimeout from 3s to 5s (part of the
cross-SDK configuration alignment). Update TestUnsetKeepsGoRedisDefaults to
expect the new default so the bump in #11306 stops failing CI.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: grant List to the generated read-only S3 identity
The chart's anvReadOnly identity only carried the Read action, so its
credentials could GetObject and HeadObject but every ListObjects request
was denied: List is a separate action and the identity check is an exact
match. Add List so the read-only credentials can list buckets and objects.
Writes stay denied. Update the README example to match.
Bump the chart to 4.47.1. The chart label is part of the s3 and
all-in-one pod templates, so the upgrade rolls the gateways and they
reload the identity config, which is only read at startup.
Fixes#11317
* helm: roll standalone S3 and all-in-one on s3 config changes
Mirror the filer checksum/s3config pod annotation in the standalone S3
and all-in-one deployments so a changed generated S3 secret triggers a
rollout during a normal helm upgrade without relying on a chart version
bump.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: populate Initiated timestamp in ListMultipartUploads
ListMultipartUploads returned each upload with only Key and UploadId,
omitting the Initiated timestamp. Clients such as GeeseFS rely on this
field to expire stale uploads and crash on its absence. Set Initiated
from the upload directory entry creation time so repeated listings
preserve the original initiation time.
* test/s3: verify Initiated timestamp in ListMultipartUploads
Add an integration test that initiates a multipart upload, lists it,
and asserts the Initiated field is populated and preserved across
repeated listings rather than reflecting the listing time.
* admin: extract isFlagExplicitlySet helper from applyViperFallback
No behavior change; the inline flag-visit check becomes a reusable
helper so the upcoming bind-address default can share it.
* admin: default to 0.0.0.0 for authenticated HTTP, keep worker gRPC on loopback
PR #11185 made the admin HTTP server default to 127.0.0.1 to stop
exposing the unauthenticated admin API on the network by accident.
That also locked out operators who already authenticated with
-adminPassword: their UI became unreachable from the network after
upgrade unless they added -ip=0.0.0.0 (see #11303).
An authenticated deployment is safe to expose, so auto-upgrade the -ip
default to 0.0.0.0 when -adminPassword or [https.admin] mTLS is
configured. The loopback default stays for the unauthenticated case, so
the unauthenticated API is never exposed on the network. An explicit
-ip is always honored.
The worker gRPC control plane has no password auth (only mTLS), so it
must not follow the HTTP upgrade. Give it a separate bind address that
stays on loopback unless -ip is explicit, so adminPassword no longer
re-exposes the unauthenticated worker stream.
* admin: hint loopback-only bind in startup banner
When the admin server binds to loopback (the default for the
unauthenticated case), print a one-line hint that it is not reachable
from other hosts and how to expose it. This helps operators who, after
the #11185 loopback default, can no longer reach the UI from another
machine quickly see the cause and the fix without reading the docs.
* admin: keep worker gRPC on loopback, decouple from https.admin mTLS
The worker gRPC auto-upgrade to 0.0.0.0 was gated on hasMTLS, which
reads the https.admin (HTTP) mTLS config. The worker gRPC mTLS comes
from grpc.admin + grpc.ca, a separate config, so:
- https.admin mTLS without grpc.admin mTLS widened the worker gRPC to
0.0.0.0 unauthenticated (re-exposing the control plane), and
- grpc.admin mTLS without https.admin mTLS left the worker gRPC on
loopback, blocking authenticated remote workers.
Drop the worker gRPC auto-upgrade entirely. The worker gRPC keeps the
raw -ip value (loopback by default), matching the pre-existing
behavior; an operator who wants remote workers sets -ip explicitly.
Only the HTTP admin listener auto-upgrades to 0.0.0.0 when
authenticated.
Addresses review feedback on #11314 from Devin and Greptile.
* s3api: add TrustedProxies allowlist helper for aws:SourceIp extraction
Introduces a policy_engine.TrustedProxies type that parses a
comma-separated list of bare IPs and CIDRs (mirroring Guard.UpdateWhiteList)
and extracts the client IP for aws:SourceIp condition evaluation.
When the direct TCP peer is in the allowlist, X-Forwarded-For is walked
right-to-left skipping trusted hops (then X-Real-Ip); otherwise the direct
peer address is returned. This is the building block for restoring
configurable forwarded-header trust removed in b88156f (#11231), as
proposed in #11302.
* s3api: honor trusted-proxy allowlist in bucket/IAM policy engine
Make ExtractConditionValuesFromRequest a method on *PolicyEngine so it
can use the engine TrustedProxies when resolving aws:SourceIp. With no
allowlist configured the behavior is unchanged from b88156f: the direct
TCP peer is used and forwarded headers are ignored. When an allowlist is
configured via SetTrustedProxies, requests from a trusted peer honor
X-Forwarded-For (right-to-left) then X-Real-Ip.
Update the two call sites (auth_credentials.go, s3api_bucket_policy_engine.go)
and the engine tests to the method form, and add a regression test for the
trusted-proxy path.
* s3api: honor trusted-proxy allowlist in IAM role/session policies
Make extractRequestContext and extractSourceIP methods on
*S3IAMIntegration so they can use the integration TrustedProxies when
resolving aws:SourceIp. With no allowlist configured the behavior is
unchanged from b88156f: the direct TCP peer is used and forwarded
headers are ignored. When an allowlist is configured via
SetTrustedProxies, requests from a trusted peer honor X-Forwarded-For
(right-to-left) then X-Real-Ip.
Update the call site in isActionExplicitlyDeniedByIAM to type-assert
the integration and use the method, and add a regression test for the
trusted-proxy path.
* s3api: load [s3.trusted_proxies] from security.toml and wire to engines
Read s3.trusted_proxies.white_list (comma-separated IPs/CIDRs) from
security.toml and propagate the allowlist to the bucket policy engine,
the IAM policy engine (persisted across rebuilds via
IdentityAccessManagement.SetTrustedProxies), and the IAM integration.
Reloaded on SIGHUP alongside the JWT signing keys. Document the new
section in the scaffold security.toml.
Closes#11302.
* s3api: harden TrustedProxies parsing and X-Forwarded-For traversal
Canonicalize bare IP entries (via net.ParseIP + String) so non-canonical
IPv6 allowlist entries such as 2001:0db8::1 match peers rendered as
2001:db8::1, and log+skip unparseable bare entries instead of storing
them inertly.
When walking X-Forwarded-For right-to-left, stop at the first malformed
(non-empty, unparseable) entry instead of skipping it, and only fall
back to the leftmost valid IP when the chain was well-formed. This
prevents a malformed hop from masking a forged IP to its left.
Addresses review feedback on #11315.
* s3api: make TrustedProxies reload race-free via atomic.Pointer
Store the trusted-proxy allowlist behind sync/atomic.Pointer in
PolicyEngine and S3IAMIntegration so SIGHUP reloads (which swap the
allowlist) cannot race with concurrent request handlers reading it.
This mirrors the existing Guard guardState pattern. The
IdentityAccessManagement copy is already protected by iam.m.
Addresses review feedback on #11315.
* shell: warn when fs.mergeVolumes source holds only orphan needles
fs.mergeVolumes traverses filer entries, so a source volume whose
needles are all orphans — filer entries lost to a crashed write or a
wiped filer store — produces only the plan header and exits 0: no move,
no skip, no error. Operators read that as a successful merge while the
real cleanup (volume.fsck) never runs, and dat>idx volumes keep coming
back read-only after restarts.
Count the source-volume needles seen during traversal and, when a plan
source was never seen but its index still reports needles, print a
warning pointing at volume.fsck. Dry-run warns too.
* shell: make needle counting concurrency-safe and count manifest sub-chunks
TraverseBfs runs its callbacks from five workers, so the plain
needlesSeen map raced between source-heavy merges (fatal concurrent
map writes). All increments now funnel through a mutex-guarded
recordSeen closure.
Manifest sub-chunks that live on planned source volumes are now
recorded too — rewriteManifestChunk visits them (including dry-run
and capacity-skipped ones) but previously never marked their source,
which produced false 'orphan needles' warnings for sources whose
chunks were all reached through manifests.
* shell: extract sourceNeedleCounter so the concurrency test covers the production path
The orphan-warning recording was a closure local to Do, so
TestWarnUnreferencedSources_ConcurrentRecording could only exercise a
test-local copy of it — a regression in the production mutex would pass
the test. Lift the map and mutex into a sourceNeedleCounter type with
record/count methods and use it from Do and the test, so the -race test
now drives the actual recording path. Trim the verbose comments added
with the warning while here.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust: apply clippy --fix to both crates
The mechanical part of a clippy sweep: `cargo clippy --all-targets --fix`
on seaweed-volume and the seaweed-worker workspace, hand-reviewed. Both
manifests declare their MSRV (1.91.1 and 1.94.1), so every suggestion
clippy applied is within it: the collapsible_if sites become let chains
(1.88, edition 2024), `% n == 0` becomes is_multiple_of (1.87),
chunks_exact with a constant becomes as_chunks (1.88), repeat().take()
becomes repeat_n (1.82), and io::Error::new(Other, ..) becomes
io::Error::other (1.74). The rest is redundant clones, borrows, casts,
closures and field names.
Nothing here changes behaviour. The three let_and_return sites in
needle_map.rs and store_ec.rs deserve a note: the `let result = ..;
result` shape was a deliberate edition-2021 workaround to drop a redb
guard before the table it borrows. Edition 2024 drops tail-expression
temporaries before locals, which is why clippy now flags it, and the
two comments that described the workaround say so instead.
Manual edits on top of the tool output: the blocks clippy rewrote are
re-indented the way rustfmt lays them out (only those blocks — the
crate is not rustfmt-clean and a whole-crate fmt would bury this diff),
the blank lines let_and_return left behind are removed, and the CRC
legacy_value test compares against a literal worked out from the
original shift formula rather than restating rotate_right.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust: clear the clippy warnings --fix cannot apply, and say why the rest stay
Hand fixes for the lints clippy only reports. Behaviour is unchanged
throughout; each rewrite is the one clippy names.
- needless_range_loop (7): index loops over shard vectors become
iterator loops. Where the old code indexed `v[..n]` the new loop
iterates `v[..n]` so an undersized vector still panics the same way.
- field_reassign_with_default (6): struct literals with `..Default`.
- redundant_pattern_matching (3): `if let Err(_) = guard.check()` becomes
`.is_err()`, which also releases the read guard at the end of the
condition instead of at the end of the block.
- manual_strip (2), manual_checked_ops, format_in_format_args,
redundant_locals, wrong_self_convention (to_vif takes self by value,
so it is into_vif; CompactEntry is Copy, so to_needle_value takes self).
- type_complexity (2): `OrphanShardLoad` and `RawNeedleEntry` name two
tuples that were spelled out inline.
- new_without_default: CompactNeedleMap gets a Default that calls new().
- suspicious_open_options: a test helper spells out `.truncate(false)`,
which is what `.create(true).write(true)` already did.
What stays, and the attribute that says so:
- too_many_arguments (10): `#[expect]` on each function. Folding 8–15
parameters into a struct is a design change, not a lint fix.
- await_holding_lock / readonly_write_lock: one test holds the store
write guard across a sleep on purpose, as a barrier that parks the
copy task at the mount block. `#[expect(.., reason = ..)]` records it.
- module_inception: needle/needle.rs mirrors the Go package layout.
Two lints become crate-wide policy in `[lints.clippy]`, with the reason
next to each: result_large_err, because every RPC path returns
tonic::Status (176 bytes) and boxing it would change every handler
signature; and needless_update, because `..Default::default()` on a
protobuf message literal is what lets a proto gain a field without
touching every constructor (all 11 sites are pb messages). The worker
workspace gets the same table and its members opt in with
`lints.workspace = true`; its generated plugin.rs also allows
large_enum_variant on prost's oneof enums.
Both crates are now clean under `cargo clippy --all-targets -- -D warnings`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust volume: use the std APIs the 1.91 MSRV already pays for
The crate declares rust-version 1.91.1, so a few things the code still
worked around are plain std now. All of them come from the 1.85–1.91
release notes; nothing here needs a newer toolchain than the manifest
already requires.
- std::sync::LazyLock (1.80) replaces the lazy_static! block in
metrics.rs, and the lazy_static dependency goes. Every use site reads
the same through Deref, so no caller changes.
- Duration::from_mins / from_hours (1.91) replace `from_secs(v * 60)`
and `from_secs(v * 3600)` in the option parser and the shard-location
refresh TTLs. One difference for the parser: an absurd count that
overflows u64 seconds now panics in release builds too, where the
multiplication used to wrap.
- Result::flatten (1.89) replaces `.and_then(|r| r)` on the replication
join handle.
- OsStr::display (1.87) replaces `to_string_lossy()` where the name was
only being formatted; the output is byte-identical.
- `#[allow]` becomes `#[expect]` (1.81) on the suppressions that are
meant to be permanent, so a suppression that stops being needed
becomes a warning rather than lingering. Doing that found four that
already had: dead_code on ChunkManifest, base_name and last_io_error,
and too_many_arguments on read_from_data_shards, which is down to
seven parameters. Those attributes are deleted. The three allows that
depend on cfg (a unix-only mutation, a linux-only field set, a
profiling-only parameter) stay as allow, because expect would be
unfulfilled on the other platforms.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* ci: add a commented-out clippy step to both Rust workflows
Both crates are warning-free under `cargo clippy --all-targets
-D warnings` now. Whether that becomes a gate is a policy call, so the
step is present but commented out; uncommenting it is the whole change.
The comment points at the `[lints.clippy]` table where crate-wide
exceptions are recorded, so the gate does not become a reason to
sprinkle allows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust volume: guard parse_duration against overflow panics
Duration::from_mins/from_hours panic when the count overflows u64
seconds. Use checked_mul so an oversized CLI value falls back to the
parser default instead of crashing volume startup.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* admin: bind worker gRPC listener to -ip instead of wildcard
The worker/plugin gRPC control plane called net.Listen("tcp", ":port")
directly, so it wildcard-bound every interface and ignored the -ip setting.
A cluster bound to loopback still exposed the unauthenticated
WorkerService/PluginControlService streams on 0.0.0.0. Bind through
util.JoinHostPort(bindIp, port) so the listener honors -ip like the
master, filer, and volume gRPC listeners.
* admin: warn when worker gRPC is exposed off loopback without mTLS
The worker gRPC stream has no password auth, so grpc.admin mTLS is the
only effective control once the listener leaves loopback. An operator who
sets -adminPassword and binds -ip=0.0.0.0 authenticates the HTTP API but
still exposes the unauthenticated worker control plane. Log a startup
warning naming the port and the mTLS knobs so the exposure is not silent.
* admin: address review on worker gRPC bind fix
- mini: reserve the admin gRPC port with util.JoinHostPort so an IPv6
bindIp (e.g. ::1) does not form an invalid unbracketed address and
lose the reservation.
- worker gRPC: track whether grpc.admin mTLS credentials actually loaded
rather than only whether they were configured, and gate the
non-loopback exposure warning on that. A cert/key that fails to load
now still warns instead of silently suppressing.
seedAndSpread() calls `volume.grow -dataNode X` without pinning the rack.
The master's grow picks the rack by weighted-random when -rack is unset,
and only one of the three racks holds the requested data node, so an
unpinned grow lands on the wrong rack two times out of three. The
VolumeGrow RPC swallows the "No matching data node" failure for
non-cache collections, so those grows count as success without creating
a volume. The per-server cap (maxGrowsPerServer=4) is then exhausted by
silent no-ops before the volumes ever spread, and seedAndSpread times
out with "volumes never spread across >=2 disks on all 3 nodes".
Pin -dataCenter dc1 and -rack rack{i} alongside -dataNode so every grow
reaches the target node. This removes the timing-sensitive assumption
that made TestECVacuumDuplicateShardClaimAcrossDisks flaky.
* rust-volume: fix Windows build of find_needle_from_ecx
The .ecx binary-search fallback path used on non-Unix targets
(Seek + Read, both &mut self receivers) requires the ecx_file
binding to be mutable. On Unix the read_exact_at path takes &self,
so the mut would be unused there — gate that warning with
#[cfg_attr(unix, allow(unused_mut))].
Without this the build-rust-volume-windows CI job fails with
E0596 at ec_volume.rs:1033, breaking the weed-volume_windows_amd64
release asset.
* rust-volume: use positional seek_read for .ecx lookups on Windows
The previous fix (making ecx_file mut) compiled but left the Windows
fallback using Seek + Read on the shared .ecx file cursor. Concurrent
find_needle_from_ecx calls could interleave seek/read and read the
wrong index entry, corrupting the binary search (raised by Devin and
Greptile review on the PR).
Switch the Windows path to std::os::windows::fs::FileExt::seek_read,
which is positional (offset passed via OVERLAPPED, cursor untouched)
and takes &self — so the binding no longer needs mut, and concurrent
callers on the cached handle can't interfere. Mirrors the existing
read_exact_at helper in storage::volume.
Add a compile_error fallback for non-unix/non-windows targets to
match the convention in storage::volume.
filer: clean up manifest resolve error propagation and add webdav test (#78)
Drop GitHub issue references from comments and trim verbose comments.
Replace the viewFromChunksOrErr helper with the existing
NonOverlappingVisibleIntervals + ViewFromVisibleIntervals at the stream
call sites, and add a WebDavFile.Read regression test for the manifest
resolution failure path.
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* filer: add guarded remote-storage client builder hook for lazy fetch
The lazy-remote fetch path (maybeLazyFetchFromRemote) resolved its
remote-storage client through the unguarded shared cache, bypassing the
SSRF chokepoint (BuildGuardedRemoteStorageClient) that the CVE-2026-73080
remediation wired into the volume, filer stream and s3 stream dial paths.
Add a RemoteStorageClientBuilder hook on Filer plus conf-only lookups on
FilerRemoteStorage, and route the lazy fetch through the builder when set
(endpoint deny-list + DNS-rebinding-safe dialer), falling back to the
shared cache otherwise. The filer server wires the builder in a follow-up.
* filer: route lazy directory listing through the guarded remote client
maybeLazyListFromRemote shared the unguarded client resolution of the
fetch path, so a caller-supplied remote endpoint was dialed without the
SSRF deny-list or rebinding-safe dialer. Resolve the conf and build the
client through buildRemoteStorageClient so the same guard covers listing.
* filer: route lazy remote delete through the guarded remote client
maybeDeleteFromRemote issued outbound DELETE/RemoveDirectory requests
through the unguarded client, giving a write-side SSRF to a caller-chosen
endpoint. Resolve the conf and build the client through
buildRemoteStorageClient so the endpoint deny-list and rebinding-safe
dialer apply to the delete path as well.
* filer server: wire the guarded remote client builder into the filer
Set Filer.BuildGuardedRemoteClient to BuildGuardedRemoteStorageClient and
forward AllowUntrustedRemoteEndpoints so the lazy-remote fetch, list and
delete paths apply the same SSRF endpoint checks as the volume and
streaming read paths.
* filer: test lazy fetch honors the guarded remote client builder
Add a regression test that sets BuildGuardedRemoteClient to a rejecting
builder and asserts maybeLazyFetchFromRemote returns no entry without
reaching the remote, covering the SSRF guard wired in the prior commits.
* filer: skip remote client for local-only lazy deletes
maybeDeleteFromRemote resolved and validated the mount's remote client
before checking entry.Remote, so a local-only file (no Remote entry) under
a mount whose endpoint the guard rejects failed to delete: the guard
error aborted the metadata deletion, leaving a file that needs no remote
operation undeletable. Move the local-only check ahead of client
construction so only remote-backed files and directories pay the guard.
* filer: build the guarded remote client inside the lazy singleflight
The lazy fetch and list paths built the guarded client before their
singleflight blocks, so concurrent requests for the same key each
allocated a fresh SDK client and HTTP transport even though only one
remote operation ran. Move client construction inside the singleflight
so the deduplicated operation builds it once, matching the per-request
guard semantics of the sibling streaming paths without the duplicate
transport churn.
* filer: test guarded rejection for the lazy list and delete paths
Add regression tests that set BuildGuardedRemoteClient to a rejecting
builder and assert the lazy list does not reach the remote, a
remote-backed file delete is blocked, and a local-only file under a
rejected mount still deletes (covering the local-only fix).
* filer: decouple lazy guarded-client build from the first caller's context
Building the guarded client inside the singleflight made concurrent
fetches share the first caller's context. If that caller canceled while
endpoint DNS validation was running, the builder returned an error and
published a not-found result to other callers whose contexts were still
valid. Build with context.WithoutCancel so the guard's DNS validation
is not tied to any single caller's cancellation, matching the list
path's existing decoupling for the remote operation itself.
* filer: reject remote-storage confs that dial blocked endpoints at load
The filer's lazy-fetch / lazy-list / remote-delete paths resolve remote
storage clients by name from FilerRemoteStorage.storageNameToConf and
dial them via remote_storage.GetRemoteStorage, which bypasses the SSRF
deny-list the volume server (BuildGuardedRemoteStorageClient) and the
filer's own direct-read path apply. A RemoteConf planted under
/etc/remote with a loopback / private / IMDS S3 endpoint is reloaded into
storageNameToConf on the next metadata-change event and then dialed on
the next cache miss — server-side request forgery from the filer.
Apply the volume server's SSRF deny-list at conf load time, the single
chokepoint that populates storageNameToConf:
- Add RemoteStorageConfValidator, injected into FilerRemoteStorage by
the filer server (the filer package cannot import the server package).
A conf that fails validation is dropped from storageNameToConf, so the
name-based client resolution on the lazy paths returns "not found"
instead of dialing the blocked endpoint.
- Add ValidateRemoteConfForLoad in weed_server, which mirrors
BuildGuardedRemoteStorageClient's gcs credential + endpoint checks
(validateRemoteEndpoint via guardedRemoteClient) without building a
client. allowUntrusted skips the check, mirroring the volume server
opt-out (-filer.allowUntrustedRemoteEndpoints).
- The filer server injects the validator at construction.
A conf whose type dials a fixed provider host (no caller-supplied
endpoint) passes; only caller-influenced endpoints are denied.
* filer: skip DNS resolution in the load-time SSRF validator
ValidateRemoteConfForLoad resolved hostnames during /etc/remote reload,
so a transient DNS failure (2s timeout) dropped the conf from the fresh
map that replaces the live map, disabling a working mount until the next
metadata event. The build-time guard (BuildGuardedRemoteStorageClient)
already re-resolves and re-validates the endpoint at dial time with the
rebinding-safe dialer, so DNS at load is redundant for security.
Split the static checks (scheme, IMDS hostnames, IP-literal blocked
addresses, gcs credentials) into validateRemoteEndpointForLoad, which
does no DNS. Hostname endpoints pass at load and are caught at dial if
they resolve to a blocked address. This preserves fail-fast for
statically-blocked confs (loopback IPs, IMDS hostnames) without letting
transient DNS failures disable mounts.
* filer: accept empty S3 endpoints in the guarded remote client builder
guardedRemoteClient returned ok=true with an empty endpoint for a
standard AWS S3 config (no custom S3Endpoint), so
BuildGuardedRemoteStorageClient and ValidateRemoteConfForLoad rejected
it with "remote endpoint is empty" — breaking standard AWS S3 mounts on
the lazy paths and the sibling streaming read paths that already use the
guarded builder.
An empty endpoint is not caller-supplied: the AWS SDK derives the
regional endpoint from the region, so there is nothing for the SSRF
guard to validate. Return ok=false for empty S3-compatible endpoints so
the builder falls through to the shared unguarded cache, matching the
historical behavior for standard AWS S3.
* filer: accept gcs credentials file paths in the guarded remote client builder
checkGcsCredentials rejected all filesystem paths, so a gcs mount
configured with remote.configure -gcs.appCredentialsFile (which stores
a path in GcsGoogleApplicationCredentials) was rejected by
BuildGuardedRemoteStorageClient with "gcs credentials must be inline
JSON". This broke existing gcs mounts on the volume, filer, and s3
remote-mount read paths that use the guarded builder.
Read and validate the file content instead of rejecting the path,
mirroring what the gcs client itself does in MakeWithHTTPClient. A path
that does not exist or does not contain valid gcs credentials is still
rejected before any client is built. guardedRemoteClient now reads the
file to extract the token exchange URL for the SSRF deny-list, so the
rebinding-safe dialer still guards the token endpoint.
* filer: resolve gcs credential paths and avoid leaking file existence
loadGcsCredentialsContent passed the raw credentials string to os.ReadFile,
so a documented ~/path (as written by remote.configure
-gcs.appCredentialsFile=~/...) was rejected because os.ReadFile does not
expand ~. It also wrapped the os.ReadFile error, which includes the
file path, exposing file existence to a caller who planted a conf with
an arbitrary path.
Resolve the path with util.ResolvePath, matching the gcs client's own
behavior in MakeWithHTTPClient. Return a generic sentinel error on read
failure so the path is not reflected in the error message. The credential
type validation still runs on the file content, so a path that does not
contain valid gcs credentials is rejected before any client is built.
* test/ec: fix EC interruption matrix slot exhaustion
The EC integration test cluster (test/erasure_coding/chaos_lifecycle_test.go)
configured each disk with -max 4 and the seedAndSpread spread loop fired
volume.grow -count 4 every 2 s with no per-server cap. Because the master
topology lags the volume.grow writes, the loop re-fired before the prior
grow was visible, over-filling disks to capacity. A full disk leaves zero
free EC shard slots (failing the cluster-wide capacity check with "no
free ec shard slots") and drops the source disk below the encode's
FreeVolumeCount >= 2 health check (failing with "no healthy replicas"),
which aborted ec.encode before any phase marker printed and made every
encode scenario in TestECInterruptionMatrix fail.
Three changes to the test cluster:
1. Raise -max from 4 to 8 per disk so the source disk always retains
FreeVolumeCount >= 2 for ec.encode's 14-shard generation (2 volume-slot
equivalents) even after the spread loop and multiple encodes.
2. Switch the spread loop from -count 4 to -count 1 so each grow lands
exactly one volume on the volume server's least-loaded disk, giving
deterministic cross-disk spreading instead of relying on a single
multi-volume grow to fan out.
3. Cap grows per server at 4 so heartbeat lag cannot run away and
over-fill disks before the master registers the prior grow.
4. Pass -minFreeSpace 0 so the test is not falsely gated by the physical
disk's free-space percentage on the host running CI (the EC shard slot
calculation separately enforces a 90 % disk-usage cap via
balancer.DiskTooFullAfter, which already guards against an over-set
maxVolumeCount on a physically full disk).
Verified locally by running TestECInterruptionMatrix twice (all encode,
decode, and balance scenarios pass, including the previously failing
encode@Deletingoriginalvolumes).
* test/ec: only count successful grows toward the spread cap
A failed volume.grow (e.g. a transient collectTopologyInfo or VolumeGrow
RPC error) would otherwise consume one of the four permitted attempts
without creating any volume, exhausting the retry budget and leaving the
loop to only poll until the Eventually timeout. Increment the per-server
counter only when commandGrow.Do returns nil.
* iceberg: confine commit/transaction/view-update write paths to authorized bucket
The create, register, and createView handlers already confine the client-
supplied metadata location to the caller table bucket and reject ".."
segments. The commit, create-on-commit, transaction, and view-update paths
read the stored metadataLocation back from the catalog and skipped the same
guard, so a location poisoned via the raw S3Tables UpdateTable API (which
persists metadataLocation verbatim) could escape the caller bucket through
a ".." segment that path.Join collapses in saveMetadataBlob.
Add confineMetadataLocation and apply it after parseS3Location on every
commit/update/transaction/view write path, mirroring the create/register/
createView check. Reject with 400 so a poisoned stored location fails the
commit instead of writing into another tenant bucket tree.
* s3tables: validate metadataLocation at the store layer
The raw S3Tables API (CreateTable, RegisterTable, UpdateTable, CreateView,
UpdateView) persisted the client-supplied metadataLocation verbatim with no
bucket-confinement or traversal check, so a caller could store a location
pointing outside its own bucket. The Iceberg REST gateway commit paths then
read that stored value back and wrote through it.
Add ValidateMetadataLocation and call it in every s3tables store handler
that accepts a metadataLocation, rejecting locations whose bucket differs
from the caller table bucket or whose path contains traversal segments. This
prevents a poisoned location from ever being persisted, complementing the
per-write-path guard added to the Iceberg commit handlers.
* iceberg/s3tables: validate location before repair and after idempotency check
Address review feedback:
- Move the commit-path confinement check ahead of repairManifests so a
poisoned stored location cannot reach manifest repair I/O before the
commit is rejected.
- Move ValidateMetadataLocation in CreateTable/CreateView to after the
existing-resource check so idempotent retries that do not consume the
requested location are not rejected for an unused bad location.
- Assert HTTP 400 in the cross-tenant reproduction tests so an unrelated
failure cannot satisfy them.
* iceberg: confine staged metadata location before load in create-on-commit
The create-on-commit path parsed the staged metadata location from the
stage-create marker and called loadMetadataFile before validating that the
staged bucket/path stay within the authorized bucket. Add the same
confineMetadataLocation guard before the read so a tampered marker cannot
direct a cross-tenant metadata read.
* iceberg/s3tables: reject bucket-only metadata locations
ValidateMetadataLocation and confineMetadataLocation accepted s3://bucket
with an empty table path. metadataDirPath then maps every such table to the
shared <TablesPath>/<bucket>/metadata directory, so tables could overwrite
or read each other's metadata files. Require a non-empty table path in both
validators; the empty-location case (where the catalog derives one) is
unaffected.
* iceberg/s3tables: reject slash-only table paths in location validation
s3://bkt/// parses to tablePath="/" which passed the empty-string check
but path.Join cleans it away, mapping to the bucket-level metadata
directory shared across tables. Update isValidTablePath to require at
least one non-empty segment and mirror the same check in
ValidateMetadataLocation, closing the gap in all callers.
* s3/lifecycle: attach admin Bearer token on internal LifecycleDelete clients
Export credential.WithS3InternalAdminAuth (renamed from withIamCacheAdminAuth)
and use it in the worker and shell lifecycle RPC adapters so lifecycle calls
carry the same admin token the IAM-cache propagation already attaches. No-op
when jwt.filer_signing.key is unset, matching the server-side checkAdminAuth.
Prepares the internal clients for the server-side auth gate that follows.
* s3/lifecycle: gate LifecycleDelete behind admin Bearer auth
Add checkAdminAuth to LifecycleDelete, matching the SeaweedS3IamCache handlers on the same internal gRPC listener (PR #11190). No-op when jwt.filer_signing.key is unset; rejects unauthenticated callers when it is. The internal worker/shell clients already attach the token in the previous commit.
* master: name the unlabeled disk layout plainly in assign errors
When no volume server serves the layout an assign targets, the error
named the empty disk type as "hdd" (HardDriveType is the empty string),
sending operators looking for servers labeled hdd when the actual
mismatch is labeled (e.g. -disk=ssd) servers versus unlabeled clients.
- describe the layout as "default (unlabeled)" when the disk type is
empty, keep %q naming for labeled types
- log the unserved-layout condition once per option instead of letting
every failing write repeat an unactionable line
Observed in production: volume servers started with -disk=ssd while CSI
mounts assign with the unlabeled layout; the per-write error stream
pointed at a nonexistent hdd fleet.
* master: bound and expire the unserved-layout warning dedupe
The dedupe map retained every distinct option key permanently. Option
keys embed request-derived fields (collection, disk type), so repeated
assignments with distinct options would grow master memory without
bound, and a retained key suppressed the warning if the same option
went unserved again after the topology recovered.
Remember last-warned timestamps instead, expiring after an hour, with a
hard cap that resets the set when a client-driven key flood fills it.
* master: silence per-retry unserved-layout log and name explicit hdd
Addresses Devin Review comments on #11290.
- The unserved-layout branch already rate-limits its warning via
assignUnservedLayoutWarning.Do, but the common epilogue still logged
lastErr at V(0) on every retry, so the flood the dedup was meant to
stop continued. Skip the epilogue log when the unserved-layout branch
owns the logging; the error is still returned to the client.
- describeDiskLayout took the canonicalized option.DiskType, but
ToDiskType folds both "" and "hdd" into HardDriveType, so an explicit
disk=hdd request was mislabeled "default (unlabeled)". Pass the
original request disk type instead: only an empty request is the
unlabeled default; an explicit hdd is named "hdd".
Adds TestAssignFailsFastNamesExplicitHdd covering the explicit-hdd
wording.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
makeupDiff called dstDatBackend.Sync() (fsync) per needle in the loop
over incrementedHasUpdatedIndexEntry. With 20000 entries in
TestLDBIndexCompaction this resulted in up to 20000 fsync calls, which
on slow CI disks exceeded the 10-minute test timeout.
Batch the sync: write all needles/tombstones first, then fsync the dat
file once in the defer alongside the existing idx fsync. The durability
guarantee is unchanged — both files are still synced before CommitCompact
writes the .cpc commit marker and swaps the files.
* cache resolved chunk manifests for Mount
* Address PR review: per-mount cache, singleflight, reuse ResolveOneChunkManifest
- Own the manifest cache per WFS mount instead of a process-global
variable, so manifests from one filer backend are never served to
another (Devin/CodeRabbit major bug).
- Coalesce concurrent cold misses via singleflight so only one fetch
runs during a cold burst (Greptile P2).
- Copy cached data after releasing the mutex so a large copy does not
block concurrent hits, inserts, and evictions (CodeRabbit nitpick).
- Reuse the existing ResolveOneChunkManifest function name instead of
introducing a new resolveOneChunkManifest wrapper.
- Validate (unmarshal) manifest bytes before caching so malformed
manifests do not poison the cache.
- Add TestChunkGroupManifestResolutionCoalescesColdMisses covering
the singleflight cold-miss path.
* Address round 2 review: coalesced-miss cancellation, test overlap
- Use singleflight.DoChan in fetchOrLoad and select on ctx.Done() so a
caller whose context is canceled while waiting for an in-flight fetch
returns ctx.Err() promptly instead of blocking for the leader's
result (Devin BUG).
- Add TestResolveOneChunkManifestCanceledWaiterReturnsDuringCoalescedMiss
covering the canceled-waiter path.
- Delay the cold-miss fixture response so the leader's fetch is still
in flight when concurrent opens join the singleflight, making the
one-fetch assertions reliable (CodeRabbit Minor).
* Address review: keep ResolveOneChunkManifest four-argument
Restore the exported ResolveOneChunkManifest to its original
four-argument signature so external callers keep compiling. Move the
cache-aware resolution into an unexported resolveOneChunkManifest
helper that accepts the per-mount ChunkManifestCache. The exported
function delegates to the helper with a nil cache, preserving the
historical uncached behavior for every non-Mount caller. The Mount
path (ChunkGroup.SetChunks) now calls the unexported helper with the
mount-owned cache. Tests and benchmarks that exercise the cache path
call the unexported helper directly.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* mount: fail reads with error when chunk manifest resolution fails
When SetChunks fails to resolve a chunk manifest (e.g. the volume is on a
remote tier with reads disabled), the sections map stays empty and
readDataAtSequential/readDataAtParallel zero-fill every missing section as
if it were a sparse hole. Reads then return all-zero data with no error,
so a plain cp of a large manifest-based file silently produces a
completely zero-filled file.
Remember the resolve error in ChunkGroup (guarded by sectionsLock) and
return it from ReadDataAt. A later successful SetChunks clears it.
Fixes the mount path of #11286.
* filer: propagate manifest resolve errors in streaming read paths
ViewFromChunks discards the chunk manifest resolve error returned by
NonOverlappingVisibleIntervals. On failure the chunk views come back
empty, and the streaming paths zero-fill the entire requested range,
serving HTTP 200 / WebDAV 200 responses whose body is all zeros.
Propagate the error in PrepareStreamContentWithThrottler,
PrepareStreamContentWithPrefetch and the WebDAV read path so these
requests fail with 500 instead.
Fixes the filer HTTP and WebDAV paths of #11286.
* mount: fail lseek with EIO when chunk manifest resolution fails
SearchChunks still consulted the stale section map after SetChunks
recorded a manifest resolution failure, so SEEK_DATA/SEEK_HOLE would
describe the unresolved regions as sparse holes or return ENXIO.
Return the recorded error from SearchChunks and map it to EIO in
Lseek.
Also add regression tests for the stream preparation error paths.
Addresses review feedback on #11287.
* storage: add Store::find_all_ec_volumes for split-disk EC lookups
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: add merge_ec_runtimes to resolve a vid's per-disk shard set
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: replace dead slots.get(14) assertion with a width-14 pin
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: build the checksum scrub plan from every per-disk runtime
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: build the local scrub plan from every per-disk runtime
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: prove the local scrub plan reaches every runtime's slots
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: make the scrub plan tests falsifiable
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: report unverifiable protection when the sidecar predates the scrubbed encode
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: commit sidecar provenance with the sidecar it describes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* volume server: scrub every disk's EC shards for CHECKSUM and LOCAL
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: run the FULL/READS parity check across split-disk shards
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* volume server: report fenced-out runtimes in FULL/READS scrubs
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: tighten verify_ec_shards ordering and missing-shard coverage
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* volume server: visit each EC volume id once in node-wide scrubs
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: cover split-disk scrub aggregation end to end
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* volume server: pin fenced-out disks and sibling-disk shards in EC scrubs
Three scrub behaviors shipped without a test at the RPC seam. Task 8
showed the seam exists, so close them here.
FULL/READS (mode 2|5) now marks a volume broken when the identity fence
excludes a runtime, where it previously reported clean. Pinned against a
control fixture whose two disks AGREE and scrub clean, so the test fails
on the clean->broken transition, not only on the message text. That needs
a structurally valid, tombstone-only .ecx (so the needle walk finds
nothing to complain about) and a seeded shard-location cache (so the
absent master does not short-circuit the scrub with an error of its own).
LOCAL (mode 3) and CHECKSUM (mode 4) now build their plans from every
per-disk runtime. Made observable by moving shard 0 -- the shard the
volume's single needle spans and the one the checksum sidecar is checked
against -- to the SIBLING disk, leaving shard 5 on the disk the singular
find_ec_volume lookup returns. Built from that disk alone, neither scrub
ever looks at shard 0.
The split-disk fixture grows a config struct rather than more positional
arguments; its defaults reproduce the existing layout byte for byte, so
the node-wide dedupe test is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: report fenced-out disks on a malformed sidecar too
`errors.extend(self.skipped)` sat below the whole status match, so only
`(Some(p), On)` ever reached it. The Invalid arm already returns a
non-empty error vector of its own, so the Go-parity contract that
silences the Off arm (`case BitrotOff: return 0, nil, nil`) does not
reach it -- appending the fence lines there costs nothing that contract
protects. A volume with BOTH a malformed sidecar and a disk the identity
fence excluded reported only the sidecar, hiding the unscanned disk
behind an unrelated integrity error.
Off stays byte-identical, and so does the `(None, On)` arm that is
documented as treating a missing payload defensively as protection off.
Off is now the ONLY status that drops the report, and the comment at the
On-path copy says so: that is the one place the parity constraint costs
us coverage.
Also corrects a false claim in the FULL/READS test's doc comment. It
said a fenced-out disk "is a disk this scrub did NOT read", which is true
only of the merge-driven parity half. The per-needle walk still resolves
`store.find_ec_volume` (store_ec.rs:281) and binds
`expected_encode_ts_ns` to that runtime (:311) -- position 0, the
EXCLUDED one on that fixture -- so `read_local_intervals`' generation
filter (:1204) makes it read the excluded disk and treat the anchor's
shards as non-local, the inverse of what `skipped` reports. The fixture's
tombstone-only .ecx walks nothing, so the test cannot tell the two apart;
the comment now says that rather than implying coverage it does not have.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: take CHECKSUM's bitrot protection from the disk that has the sidecar
`EcChecksumScrubPlan::for_volumes` read `(prot, status)` off the ANCHOR.
The anchor is the first shard-bearing runtime at the maximum `encode_ts_ns`,
chosen with no regard for which disk holds the `.ecsum`.
That sidecar is deliberately NOT mirrored across disks -- `ec_metadata_dirs()`
exists so one authoritative copy stays reachable rather than being duplicated
-- and at mount `EcVolume::new` resolves it via `load_active_bitrot_sidecar(&[])`
with no sibling directories at all; only the `VolumeEcShardsMount` RPC ever
passes `ec_metadata_dirs()`. So after EVERY volume-server restart, the
split-disk runtime that does not physically hold the sidecar mounts
`BitrotStatus::Off`. When the one copy lives on disk 1 and the anchor is disk 0,
`run()` hit `case BitrotOff` and returned `(0, [], [])`: the whole volume
scrubbed clean, silently. That is the steady state for roughly half of all
mirrored split-disk layouts, and it is the exact failure this branch exists to
remove.
Source protection from the first MERGED runtime that has any -- `On` if one
does, else `Invalid`, else the anchor's `Off`. Two facts make that safe, and
both are load-bearing:
- Every runtime that mounted `On` already passed the `geometry_matches` gate
in `load_bitrot_for_generation`, so its manifest agrees with the volume's
layout. A sidecar that contradicted it would have failed the mount.
- All merged runtimes share the same `encode_ts_ns` by construction of the
identity fence, so a sidecar from any of them describes the same encode run.
The `unverifiable_sidecar` provenance rule four lines down read
`anchor.bitrot_source_dir`; it now reads the SAME runtime `prot` came from.
Otherwise the two would describe different sidecars and the rule would vouch
for a manifest nobody is scanning against. One consequence worth naming: that
source dir is now non-empty by construction (a runtime with protection found a
file), where the anchor's was often "" and short-circuited the rule -- so on a
fenced volume whose anchor had no sidecar, an unverifiable-protection note now
surfaces where previously nothing was reported at all.
`run()` is untouched, and the `BitrotStatus::Off` arm still returns
`(0, [], [])` exactly, for Go parity with `case BitrotOff: return 0, nil, nil`.
`parity_shards` still comes from the anchor while `prot` may come from a
sibling; the geometry gate above makes them agree, and slot-width agreement is
handled separately.
The test drives mode 4 through the real RPC against a split-disk volume whose
sidecar exists only on dir1, and asserts up front that the anchor mounted `Off`
and the sibling `On` -- otherwise it would prove nothing. Reverting this commit's
one-line source change makes it report `[]` instead of `[0, 5]`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: pin the slot width, contain the shard-size fallback, and cover multi-disk FULL
Five findings from the whole-branch review, none of which changes what a
healthy volume reports.
Slot width was undefined and the two consumers disagreed (ec_volume.rs).
`merge_ec_runtimes` sizes `slots` to the WIDEST merged runtime, but the identity
fence keys on `encode_ts_ns` alone and never on geometry -- so two
same-generation runtimes whose `.vif`s disagree do merge. The mode 2|5 arm
truncates to the anchor's `data+parity` and silently drops the surplus slots,
while `EcChecksumScrubPlan::for_volumes` iterated the full width and emitted
"present but missing from sidecar manifest" for exactly those ids. Nothing in
the volume describes them -- the sidecar manifest and the Reed-Solomon matrix
are both the anchor's -- so that message was the width disagreement talking, not
a finding. The `slots` field doc now states the contract (the range is the
anchor's geometry; every consumer truncates to it) and CHECKSUM truncates.
The LOCAL `shard_size` fallback had grown a node-wide blast radius
(ec_volume.rs). `anchor.shard_file_size()` returns the anchor's FIRST held
shard, not a maximum. Before aggregation the plan read only that runtime's own
shards, so a truncated shard was contained to its disk; now that one value sizes
every merged sibling's shards, mis-offsetting `locate_data` and manufacturing
needle corruption across the node. Take the max over the merged slots, which is
how `verify_ec_shards` already answers the same question
(`if size > shard_size { shard_size = size }`). Only on the legacy
`dat_file_size == 0` path.
Multi-disk `all_local` had no end-to-end test (grpc_server.rs). The parity check
is gated on every shard being present, and the one all-local fixture keeps them
in a single directory, so every entry of `dirs` is the same string and a
permutation or off-by-one in the `slots` -> `dirs` mapping is invisible;
`test_verify_ec_shards_reads_shards_from_multiple_dirs` builds its `dirs` by
hand and never goes through `merge_ec_runtimes`. The new fixture is a real 10+4
encode split 0..=6 / 7..=13 across two store locations (the `.dat`/`.idx` stay
outside both, so `prune_incomplete_ec_with_sibling_dat` has nothing to act on),
driven through the real RPC: clean first, then a corrupted PARITY shard on the
SECOND disk -- which only the parity half can see, and only through a correct
mapping. Shifting that mapping by one, or computing `all_local` from the anchor
alone, both make it report `[]` instead of `[13]`.
Deleted `test_ec_volume_enumeration_is_deduped` (store_ec_reconcile.rs). It
built `raw` from `store.locations` and then applied its OWN inline
`filter(|v| seen.insert(*v))`, asserting on that -- a property of
`HashSet::insert`, never reaching the production dedupe. That path is covered by
`test_scrub_ec_volume_node_wide_dedupes_a_split_disk_volume`, which does fail
(2 != 1) when the dedupe is removed.
Corrected `test_verify_ec_shards_treats_a_none_dir_as_missing`'s docstring
(ec_encoder.rs). It claimed the unmounted shard "must not drag the shards that
ARE mounted down with it", but `dirs[5] = None` puts shard 5 in `broken_shards`
before the block loop, so every iteration takes the `read_failed` arm and the
parity comparison never runs: corrupting a mounted shard in that fixture changes
nothing about the result. The assertions are unchanged; the docstring now states
what they actually establish.
Also refreshed two comments that cited `shard_file_size() - 1` as the reason
`merge_ec_runtimes` prefers a shard-bearing anchor -- true before this commit,
stale after it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: correct the Fix 1 rationale and truncate the shard-size scan
The safety argument attached to `EcChecksumScrubPlan::for_volumes`'s protection
selection was false as written, and it is the argument a reviewer reads first.
`geometry_matches` compares a sidecar against the MOUNTING runtime's own
data/parity/block size, not the anchor's, and returns true vacuously when
`ec_shard_config` is `None` -- so it establishes agreement only when all merged
runtimes share one geometry, which an `encode_ts_ns`-only fence does not
guarantee and which `test_checksum_scrub_truncates_slots_to_the_anchors_geometry`
constructs a counterexample to. The second clause was weaker than stated too: a
`.ecsum` records no encode identity at all, so merged runtimes agreeing on
`encode_ts_ns` does not transfer to the sidecar.
Replace it with the property that is true, checkable from the selection itself,
and stronger for what actually matters. `anchor` is an element of `merged`, so
the `.unwrap_or(anchor)` fallback is reached only when no merged runtime is `On`
and none is `Invalid` -- in which case the anchor is necessarily `Off`. The
status can therefore only move `Off -> On`, `Off -> Invalid` or
`Invalid -> On`; never `On -> Off`, never `Invalid -> Off`. This selection
cannot stop a volume that was being scanned from being scanned, and cannot turn
a reported integrity error into silence: every change it makes is toward more
verification. The comment now also states what it does NOT establish -- geometry
agreement is not guaranteed -- and names geometry fencing as the follow-up that
would close it.
Second, `EcLocalScrubPlan::for_volumes`'s `shard_size` max scanned the FULL slot
width, violating the `slots` contract documented in the same commit that
introduced the max: the volume's shard-id range is the anchor's geometry and
every consumer must truncate to it. Pre-fix that input could not exist, because
`anchor.shard_file_size()` read only the anchor's own anchor-sized vector -- so
the max opened a new, narrow path to the same node-wide mis-sizing it exists to
close (same-generation runtimes with disagreeing `.vif`s, the wider one holding
an out-of-geometry shard larger than the in-geometry ones, `dat_file_size == 0`).
`.take(anchor.data_shards + anchor.parity_shards)` mirrors the truncation
already applied to the CHECKSUM shard scan.
The sibling `shards:` vector is left untruncated on purpose: every access in
`EcLocalScrubPlan::run` is `shards.get(sid)` with `sid < data_shards`, so the
surplus entries are inert.
No behavior change for any healthy volume, and no test added -- the suite is
unchanged at 575 passing, 0 failing, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: aggregate split-disk runtimes in Go scrubs, mirroring Rust
Go volume scrubs previously used FindEcVolume (first runtime only), so a
volume whose EC shards are split across multiple disks was scrubbed against
just one disk's shards and the others were silently skipped. Node-wide
ScrubEcVolume also appended each disk's EcVolumeIds without deduplication,
scrubbing a split-disk volume once per disk.
Add MergedEcRuntimes/MergeEcRuntimes (Go counterpart to Rust's
merge_ec_runtimes): select the maximum EncodeTsNs as the anchor generation,
fence out runtimes whose encode generation or geometry (DataShards,
ParityShards, BlockSize) disagrees with the anchor, merge shard handles by
shard ID, and report excluded runtimes rather than dropping them. Wire it
into every scrub mode:
- INDEX: scrub the anchor's index, report skipped runtimes.
- LOCAL: aggregate local shards across all merged runtimes via a synthetic
EcVolume built from the merged shard slots.
- FULL/READS: resolve the runtime matching the anchor's encode generation
(not the first match) so the needle walk and parity phase inspect one
encode run; report skipped runtimes.
- CHECKSUM: take bitrot protection from the first merged runtime that has a
valid sidecar (On, else Invalid, else anchor's Off), preserve invalid
sidecar errors from every other merged runtime, and report skipped
runtimes.
Deduplicate EC volume IDs in node-wide ScrubEcVolume so each volume is
scrubbed exactly once.
Refactor ScrubEcVolume to share the per-needle walk via scrubEcVolumeWalk,
called by both the legacy first-runtime path and the new merged path.
Add Go regression tests covering split-disk deduplication, encode-generation
fencing, geometry fencing, sibling-disk LOCAL reach, and merge anchor
selection.
Rust: keep the previously-landed merge/fence/checksum changes intact; revert
incidental cargo-fmt drift from unrelated files so the diff stays focused.
* ec: fence merged CHECKSUM on sidecar encode generation and fix legacy shard size
Address two review findings on the Go merged-runtime scrub:
1. Sidecar provenance: a merged runtime can load a bitrot sidecar from a
sibling metadata directory (ReloadBitrotSidecar), and the merge fence may
then exclude the runtime owning that directory. Generation-0 sidecars do
not identify the encode run, so geometry validation alone cannot prove the
borrowed manifest describes the anchor shards. If the sidecar records a
non-zero EncodeTsNs that disagrees with the anchor, refuse the scan
instead of applying stale checksums to current shards and reporting false
corruption.
2. Legacy shard size: for volumes without datFileSize in .vif,
LocateEcShardNeedleInterval derives the shard size from Shards[0].ecdFileSize.
The merged shard set is compacted in shard-ID order, so a truncated
lowest-ID shard would shrink every interval and misread intact sibling
shards. Synthesize a datFileSize from the maximum mounted shard size when
the anchor lacks one, so the datFileSize>0 path uses the largest shard
size across all merged runtimes.
* ec: fix copylocks, legacy shard boundary, and encode-aware Rust lookups
Address review findings from CodeRabbit and Devin:
Go (ec_volume_merge.go):
- Remove bitrotLock copy from the synthetic EcVolume: copying a sync.RWMutex
is a go vet copylocks error. The synthetic volume uses its own zero-value
mutex; bitrot/bitrotStatus are set directly before ChecksumScrub reads them
via BitrotProtection(), so no concurrent access occurs.
- Fix legacy shard-size boundary: synthesize datFileSize from
(maxShardSize - 1) * DataShards, not maxShardSize * DataShards, to match
the legacy fallback in LocateEcShardNeedleInterval (ecdFileSize - 1). An
exact large-block boundary is ambiguous; the unadjusted size would select
an extra large row and misread intact sibling shards.
Rust (store_ec.rs):
- Add find_ec_volume_for_scrub helper that resolves by encode generation
(not first-match find_ec_volume) and use it in scrub_snapshot_under_lock,
write_back_shard_locations, and the post-refresh shard-location read.
Previously the encode-aware lookup was only used for the initial runtime
selection; the cache write-back and per-needle snapshot still used
first-match, so a split-disk volume whose first runtime was from an older
encode run would write to and read from the wrong runtime's shard-location
cache and falsely abort with 'remounted as a different encode run'.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* test(fuse_failover): dump chunk list and hex on append corruption
The failover append test (TestAppendWhileVolumeServerRestarts) failed
in CI with an 8-byte NUL region at offset 632 that appeared in both
the writer mount and the filer own view, but the failure message
only showed a quoted-string window around the divergence. That is
not enough to tell which chunk covered the zeroed bytes or which
volume server held it, so the next recurrence would be just as
unattributable.
Add a FileChunkList helper that reads the filer resolved chunk
list, and on failure dump:
- every chunk fid, offset, size, volume id, and current master
holders, flagging the chunk that covers the first divergence;
- a hex+ASCII dump of the writer mount around the divergence so
the exact zero-filled region is visible byte-for-byte.
No production code is touched; this only makes the test fail louder.
* test(fuse_failover): preserve diagnostic collection errors
Address review feedback from CodeRabbit and Greptile on PR #11285:
- FileChunkList now returns the wrapped ParseUint error when
fid.volume_id is zero and the file_id prefix is invalid, matching
FileVolumeIds instead of silently keeping vid=0 (which would
query /dir/lookup?volumeId=0 and report the wrong holders).
- dumpChunkList captures the VolumeHolders error and renders it as
'lookup failed: ...' so a failed master request is distinguishable
from a successful lookup with no holders (both previously showed
'holders=[unknown]').
- runChaosAppend captures the writer-mount read error and includes
it in the failure message so an unavailable writer view is not
mistaken for corrupted content.
MasterServer.Shutdown only stopped the Hashicorp raft implementation;
when using the default goraft backend, the raft event-loop goroutine
(leaderLoop/followerLoop) kept running after the master shut down. In
the in-process test harness this leaked goroutines across sequential
test runs, and a stale event occasionally reached a leader at term 0
and tripped the goraft "leader.elected.at.same.term" assertion,
crashing the whole test binary (CI run 34670959967, PR 11279).
Stop the goraft server in Shutdown() so its goroutines exit cleanly,
and bump seaweedfs/raft to v1.2.1 which replaces that assertion with a
graceful step-down to Follower instead of a panic.
* s3: reject virtual-host bucket retargeting via X-Forwarded-Host
SigV4 verification tries the client-supplied X-Forwarded-Host as a
signed host candidate, while routing and IAM select the bucket from
the actual Host header. A presigned URL for one virtual-host bucket
could therefore be retargeted to another bucket accessible to the same
signing identity by changing Host and adding X-Forwarded-Host.
After the signature matches a host candidate, extract the bucket that
the candidate implies (via the configured virtual-host domains) and
compare it with the bucket the router selected. Reject when they
differ, before returning success.
* test(s3api): cover virtual-host presigned URL retargeting
Add unit tests for bucketFromVirtualHost and end-to-end tests that
reproduce the X-Forwarded-Host retargeting attack for both presigned
and signed requests, plus a negative test confirming the legitimate
same-bucket case still verifies.
* s3: harden bucketFromVirtualHost for case and overlapping domains
Compare host and domain suffixes case-insensitively so a mixed-case
X-Forwarded-Host cannot bypass the consistency check. Only treat the
exact path-style domain as non-virtual-host; subdomains of a path-style
domain still match the virtual-host router pattern and must be checked.
* s3: reject list-type paired with another operation subresource
?list-type=2&ownershipControls= routes to ListObjectsV2 (the list-type
route is registered first) while the IAM action resolver resolves the
ownershipControls selector to s3:GetBucketOwnershipControls. A principal
denied s3:ListBucket but allowed s3:GetBucketOwnershipControls would
therefore list the bucket. list-type selects an operation just like the
other keys in operationSubresources, so add it there and reject the
combination before routing, matching the fix for policy&tagging (#10987).
* s3: resolve list-type to s3:ListBucket ahead of bucket subresources
The router registers the ListObjectsV2 route ahead of the bucket
subresource routes, so the action resolver should resolve list-type the
same way. Without this, a request carrying list-type and another operation
selector resolves to the subresource action (e.g. s3:GetBucketOwnershipControls)
while being served by ListObjectsV2. The ambiguity guard rejects such
combinations before routing, but resolving list-type to s3:ListBucket keeps
the resolver aligned with the router, mirroring how versions is handled.
* s3: match list-type=2 exactly in action resolver
The router selects ListObjectsV2 only for list-type=2; other values fall
through to the subresource routes. Resolve the same way so the action
matches the handler for every list-type value, not just 2.
* feat(s3api): add bucket quota S3 extension via ?seaweedfs-quota
Add a SeaweedFS-specific S3 subresource for bucket quota management:
PUT /{bucket}?seaweedfs-quota — set bucket quota (s3:PutBucketQuota)
GET /{bucket}?seaweedfs-quota — get bucket quota (s3:GetBucketQuota)
The request/response body is JSON:
{"quota_size": 100, "quota_unit": "GB", "quota_enabled": true}
Quota is stored on the bucket's filer entry (positive = enabled,
negative = disabled but retained, zero = no quota), matching the
existing admin REST API behavior. When quota is cleared, the bucket's
read-only flag is also lifted.
Authentication uses the existing S3 SigV4 flow — no new global secret
is needed. Authorization uses two new dedicated IAM permissions:
s3:PutBucketQuota
s3:GetBucketQuota
This allows integrations like Apache CloudStack to manage per-bucket
quotas through the S3 endpoint with a scoped credential, without
exposing the broad admin REST API or requiring a separate admin token.
The credential can be limited to s3:PutBucketQuota/s3:GetBucketQuota
only, preventing bucket deletion, user management, or cluster topology
changes.
The coarse-grained ACTION_PUT_BUCKET_QUOTA/ACTION_GET_BUCKET_QUOTA
constants are added to s3_constants, and the action resolver maps the
seaweedfs-quota query parameter to the fine-grained s3: actions for
policy evaluation.
* docs: update design for S3 ?seaweedfs-quota extension approach
Replace the broad admin REST API + bearer-token design with the narrow,
scoped S3 ?seaweedfs-quota extension. Update quota, usage reporting, and
SeaweedFS-side changes sections to reflect PR #11279.
* fix(s3api): address review comments on quota handler
Fix four issues identified by Devin, Greptile, and CodeRabbit reviews:
1. Integer overflow in convertQuotaToBytes: large quota_size values
(e.g. 8388608 TB) could overflow int64, wrapping to negative and
being silently treated as zero quota. Now returns an error when
size * multiplier would exceed math.MaxInt64.
2. Disabled quotas returned negative sizes in GET: the GET handler
returned entry.Quota directly, which is negative for disabled-but-
retained quotas. Now returns the absolute magnitude as quota_size
and derives quota_enabled from the sign, making the response
round-trippable.
3. Missing buckets returned 500 instead of NoSuchBucket: the PUT
handler treated all lookup failures as internal errors. Now
distinguishes filer_pb.ErrNotFound and returns ErrNoSuchBucket.
4. Trailing JSON was silently accepted: the decoder read only the
first JSON object without checking for trailing data. Now
requires EOF after the object, rejecting malformed payloads.
Also add tests for overflow detection and trailing data rejection.
* fix(s3api): cast math.MaxInt64 to int64 for 32-bit vet
On 32-bit platforms, math.MaxInt64 is an untyped int constant that
overflows int (32-bit) when used directly in fmt.Errorf with %d.
Cast to int64 explicitly to fix Go Vet 32-bit.
* docs: reconcile design doc with implementation and add AWS tools note
- Resolve open question about IAM endpoint path: driver accepts optional
iamUrl and defaults to <s3Url>/iam
- Add note explaining ?seaweedfs-quota is not callable by standard AWS tools
(aws s3api, s3cmd, rclone), and how this compares to MinIO and Ceph quota
APIs which also live outside the standard S3 API
* docs: fix IAM endpoint default — SeaweedFS IAM is at POST / on S3 endpoint
SeaweedFS registers its embedded IAM API at POST / on the same S3
endpoint (UnifiedPostHandler), not under /iam. The design doc
previously said the driver defaults iamUrl to <s3Url>/iam, which would
send IAM operations to an unregistered path. Correct the default to
s3Url.
Found by Greptile review on PR #11279.
* docs: fix credential model, signer, and GET response shape in design doc
Three issues found by CodeRabbit review on PR #11279:
1. Credential-scope contradiction: the doc claimed the service credential
is scoped to only s3:PutBucketQuota/s3:GetBucketQuota, but the
implementation uses it as the admin credential for all operations
(bucket CRUD, IAM user provisioning, quota). Document the actual
model.
2. S3Signer -> AWSS3V4Signer: the doc said 'S3Signer for SigV4 signing'
but S3Signer is legacy SigV2. Correct to AWSS3V4Signer.
3. GET response shape: the doc showed a single JSON example with 'GB'
for both PUT and GET, but GET always returns quota_unit 'B' and the
absolute byte count. Document PUT input and GET response separately.
* s3: support StringEqualsIgnoreCase and related condition operators
The S3 bucket-policy condition engine rejected StringEqualsIgnoreCase
(and StringNotEqualsIgnoreCase, StringLikeIgnoreCase,
StringNotLikeIgnoreCase), which AWS and the IAM policy engine both
accept. Add evaluators and register them in GetConditionEvaluator so
valid policies using these operators evaluate correctly instead of
being skipped.
* s3: reject bucket policies with unsupported condition operators
validateStatement did not check Condition operators, so a policy with
an unknown operator (e.g. a typo or unsupported key) was accepted at
upload time and only surfaced at evaluation, where it was silently
skipped. Reuse GetConditionEvaluator to reject unknown operators when
a policy is parsed or stored, failing closed at the entry point
instead of relying on evaluation-time handling.
* s3: fail closed on unsupported condition operators at evaluation
EvaluateConditions skipped statements whose condition operator was
unsupported, logging a warning and continuing. With no remaining
conditions to fail, the function returned true, so an Allow statement
conditioned on an unrecognized operator became unconditional and
granted access to private objects. Return false instead so an
unrecognized operator fails the condition block and the statement does
not match, matching the fail-closed behavior of the IAM policy engine.
* s3: validate condition operators at upload time only, not load time
Validating condition operators in validateStatement rejected the whole
policy document from ParsePolicy, which SetBucketPolicy uses when loading
stored bucket policies. A legacy policy saved before this change could
contain an unsupported operator, and rejecting it at load time dropped
the entire policy - including unrelated explicit Deny statements - so
the bucket lost its protections. Move the operator check into
ValidateBucketPolicy, which only the PutBucketPolicy handler and admin
UI run at upload time, so legacy policies still load and EvaluateConditions
fails the unsupported statement closed instead.
* s3: drop non-AWS StringLikeIgnoreCase and StringNotLikeIgnoreCase operators
AWS defines StringEqualsIgnoreCase and StringNotEqualsIgnoreCase but
not StringLikeIgnoreCase or StringNotLikeIgnoreCase (StringLike and
StringNotLike are case-sensitive only). Registering the wildcard
IgnoreCase variants made the engine accept operators AWS rejects. Keep
only the two AWS-defined IgnoreCase operators and add a test asserting
the wildcard IgnoreCase names are unsupported.
* s3 sink: report the source read error the SDK hides
filer.backup stops for good on an event whose chunks are gone from the
volume servers: the uploader reads the body, the read fails with the
volume's 404, and the AWS SDK returns "ContentLength=N with Body length 0"
without the cause. isIgnorable404 would skip such an event, but it never
sees the 404, so the event is retried forever and the checkpoint never
advances.
ChunkStreamReader keeps its first source failure and the s3 sink returns
it when the upload fails.
* s3 sink: trim verbose comments on source error propagation
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(vacuum): stop comparing compact size against the live needle map
CompactByIndex's post-copy integrity check compared bytes written to
the .cpd against v.nm.ContentSize()-DeletedSize(), the live map that
keeps mutating for as long as the volume stays writable during the
copy. Any write landing after the point-in-time index snapshot was
loaded made the live map's tally exceed what got copied, aborting
compaction with "unexpected new data size" — even though
CommitCompact's makeupDiff exists specifically to reconcile writes
that land mid-copy. On a busy volume this can fail every vacuum cycle.
Tally the expected live size from oldNm, the same frozen snapshot the
copy loop reads from, instead of the live map. This keeps the check's
original protection (destination smaller than what should have been
copied signals real data loss) while removing the false positive from
ordinary concurrent traffic.
* fix(vacuum): stop double-subtracting skipped bytes from the size check
Unreadable needles return before reaching the expectedLiveBytes tally,
so it already excludes them. Subtracting skippedDataBytes again on top
loosened the integrity check's margin by that same amount, letting a
.cpd short of the true expected size slip past undetected — the exact
failure mode the check exists to catch. Flagged independently by three
automated PR reviewers (Devin, Greptile, CodeRabbit).
Extract the comparison into exceedsExpectedCompactedSize and drop the
subtraction entirely; add TestExceedsExpectedCompactedSize to pin the
threshold to expectedLiveBytes alone.
* fix(vacuum): trim verbose integrity-check comment
Reduce the 8-line block comment to a concise 3-line rationale. No
behavior change.
* fix(vacuum): mirror compact integrity check in Rust volume server
Mirror the Go fix in the Rust volume server's do_compact_by_index:
tally expected_live_bytes from the frozen index snapshot (not the live
needle map) and compare the compacted .dat against it after the copy.
Unreadable needles already return before the tally, so no skipped-byte
adjustment is needed. Adds exceeds_expected_compacted_size and two
regression tests.
* fix(vacuum): exercise makeup_diff in Rust concurrent-write test
Address CodeRabbit review: write a needle after compaction (before
commit), then call commit_compact() and assert the late write survives
via makeup_diff. This actually exercises the concurrent-write path
rather than just confirming the integrity check passes.
---------
Co-authored-by: chrislusf <chris.lu@gmail.com>
* test: reproduce expired directory cache degrading lookup to N RPCs
After cacheMetaTtlSec elapses the kernel can still serve a directory
listing from its page cache, so ReadDir never runs and EnsureVisited is
not called. Metadata lookups then fall through to one LookupEntry RPC
per entry instead of rebuilding the directory cache once.
Issue #11262
* mount: add expired-directory rebuild predicate with cooldown to InodeToPath
ShouldRebuildExpiredDir distinguishes a TTL-expired cached directory from
a never-cached, invalidated, evicted, or read-through one (those clear
isChildrenCached, while a plain TTL expiry keeps it set). It also gates
retries on a cooldown since the last failed rebuild attempt, recorded by
MarkRebuildAttempt, so a transient listing failure does not trigger a
full rebuild on every later lookup.
Issue #11262
* mount: rebuild expired directory cache on entry lookup
When the kernel still serves a directory listing from its page cache past
cacheMetaTtlSec, ReadDir never runs and EnsureVisited is not called, so
lookupEntry issues one LookupEntry RPC per entry. Rebuild the expired
directory once via ensureDirectoryVisited before the cache-hit check so
later lookups are served locally. The EnsureVisited singleflight
deduplicates concurrent rebuilds.
On a non-oversized rebuild failure, record the attempt so the cooldown
suppresses repeated rebuilds while the listing keeps failing; once it
elapses a later lookup retries, recovering without waiting for ReadDir.
Oversized dirs are already marked read-through by ensureDirectoryVisited.
Issue #11262
* test: cover concurrent rebuild dedup and rebuild-cooldown fallback
Add a test that runs concurrent lookups into the same expired directory
behind a gated listing, asserting they share one rebuild via the
EnsureVisited singleflight. Add a test that a failed rebuild records the
attempt so an immediate retry is suppressed (per-entry RPC fallback), and
that once the cooldown elapses and the filer recovers a later lookup
rebuilds the cache.
Issue #11262
* mount: wait for pending async flush before rebuilding parent cache
The rebuild lists the parent directory from the filer, so a pending
async flush of the target entry must land first; otherwise the rebuilt
cache captures pre-flush metadata and the cache-hit path returns it
without the wait that guards the filer-fallback path. waitForPendingAsync
Flush is a no-op when no flush is pending, so the common case is unaffected.
Issue #11262
* Add ResolveIdentityClaim helper for OIDC audit identity
ComputeParentUser derives a stable per-identity hash from (sub, iss) for
internal keying, but it is opaque and not human-readable. Audit logs for
STS-assumed OIDC sessions currently surface that opaque value (or the
random session id) as the requester, leaving no authoritative trace of the
federated user.
Add ResolveIdentityClaim next to ComputeParentUser to recover a
human-readable, server-asserted identity attribute from the STS request
context populated at federation time. It walks a priority list
(preferred_username, email, name, sub) so a federated session always
audits against a stable OIDC claim rather than a client-supplied role
session name.
For #11264
* Surface authoritative OIDC identity claim in S3 audit log
For STS-assumed sessions minted from an OIDC web identity, the audit log
requester field is the opaque session subject, which cannot be traced back
to the federated user who performed the operation. The OIDC identity claims
(preferred_username, email, sub) are already carried in the session request
context and reach the auth layer as identity.Claims, but they were never
surfaced to the audit log.
Add a requester_identity field to the S3 access audit log, populated from
the authoritative OIDC identity claim resolved via ResolveIdentityClaim.
The claim is propagated through the shared identity holder (the same
mechanism the requester name and principal ARN already use) so it survives
the request-context copy that hides auth-set values from the outer audit
middleware.
The existing requester field is left unchanged for backward compatibility;
requester_identity is empty for non-federated sessions, where requester
already carries the real username.
For #11264
* Gate OIDC audit identity on federation marker and harden resolver
Address review feedback (Devin Review, Greptile) on the initial
implementation:
- Non-federated STS sessions no longer gain a false requester_identity.
ValidateJWTWithClaims merges the JWT registered sub claim (the opaque
session id) into RequestContext for sessions without an explicit request
context, so the previous ResolveIdentityClaim fallback to sub surfaced
that session id as an authoritative identity. Resolution is now gated on
SessionInfo.ParentUser, which is set only for OIDC-federated sessions in
AssumeRoleWithWebIdentity. The claim is resolved from the original
sessionInfo.RequestContext (not the local claims map, whose sub the bearer
path overwrites with the session subject) so SigV4 and bearer sessions
surface the same identity.
- ResolveIdentityClaim now trims whitespace and treats whitespace-only
claims as absent, so a blank preferred_username no longer masks a usable
email or sub.
The resolved claim is carried on Identity.IdentityClaim (and IAMIdentity for
the bearer path) rather than re-derived in recordIdentityInContext, making
the federation gate explicit at the auth boundary.
For #11264
* Resolve OIDC identity claim for external bearer tokens
The external OIDC bearer path (a raw OIDC JWT presented directly, not via
STS) populates Claims with preferred_username/email/name/sub from the
validated token but did not set IdentityClaim, so requester_identity stayed
blank for that authentication path. Resolve the claim there too — sub is the
real OIDC subject on this path (not an STS session id), so no federation
gate is needed.
Also drop an ineffectual ctx assignment flagged by ineffassign in the audit
test.
For #11264
* refactor: extract update event handling into processUpdateEvent
Pull the OldEntry/NewEntry update branch of the remote sync event
processor into its own function so the rename skip logic can be
exercised by tests with stub clients. No behavior change.
* test: reproduce remote sync rename dropping the new key
A rename under a remote mount arrives as an update whose NewEntry
inherits the source RemoteEntry. shouldSendToRemote returns false
for it, so processUpdateEvent skipped the event without writing the
new key, while the filer had already deleted the old object. The test
runs such an event through processUpdateEvent and expects both a
delete of the old key and a write of the new one. Fails before the
fix. See #11261.
* fix: write the new key when a remote-synced file is renamed
A rename under a remote mount arrives as an update whose NewEntry
inherits the source RemoteEntry, so shouldSendToRemote returns false
(RemoteMtime >= Mtime) and processUpdateEvent skipped the event.
That skip is only valid when the destination key is unchanged; a path
change always needs a write, and the delete-old/write-new handling
below the early return is exactly what a rename needs. Guard the skip
with proto.Equal(oldDest, dest) so a rename falls through to it.
Fixes#11261.
* fix: skip empty upload when renaming a remote-only entry
A remote-only entry (no local chunks or content, data lives only on
the remote object) carries a positive RemoteSize but nothing for
NewFileReader to read. After the previous commit lets a rename fall
through to the delete-old/write-new path, such a rename would upload
EOF and create a zero-byte object at the new key, then stamp it as
synced. Guard the write so a path change on a remote-only entry skips
the upload instead of replacing the file with zero bytes. The filer
has already deleted the old object, so the data is gone regardless;
this avoids leaving a misleading empty object behind.
* fix: propagate old-key delete errors except already-deleted
When deleting the old key on a rename fails for a non-multipart entry,
the error was swallowed and the write proceeded, which could leave both
remote keys. Return the error so MetadataProcessor retries the event.
The filer deletes the source remote object synchronously during the
rename, so the sync delete is redundant and the object may already be
gone. GCS reports that as ErrRemoteObjectNotFound (unlike S3/Azure,
whose deletes are idempotent), so treat it as a successful deletion and
continue to retriedWriteFile rather than pinning the sync offset.
* fix(s3api/iam): fail config snapshot on empty or malformed IAM files
A full IAM reload reads every identity/policy/service-account/group file
from the filer. When an external secrets tool rewrites a file, a reload
that reads it mid-rewrite sees empty or partially-written content. The
identity, policy and service-account loaders silently skipped such files
(``continue``), so the snapshot was missing entries that still existed
on disk. The atomic swap then installed an incomplete identity set while
``isAuthEnabled`` stayed on, denying unrelated clients mid-reload
(#11259).
The group loader and the read-error paths already fail the snapshot in
this situation (a skipped entry reads as deleted). Apply the same
behavior to empty content and unmarshal failures across the identity,
policy, service-account and group loaders, so a transient mid-rewrite
fails the reload (preserving the last known-good state) instead of
silently dropping entries.
* fix(s3api/iam): coalesce burst IAM config reloads through the reload queue
onIamConfigChange did a full synchronous reload for every identity/policy
file change event. When several independently-refreshing credentials
rewrite their files within the same second, that produced a burst of
dozens of back-to-back full reloads, each reading the whole store and
widening the window where a mid-rewrite file is observed (#11259).
Route every IAM config change through the existing coalescing reload
queue (scheduleReload/reloadRetryLoop) instead. A burst of N events now
collapses into a single reload (plus one tail reload for events that
arrived while one was in flight). scheduleReload gains a reason argument
for the existing log line; the reloadRetryLoop already retries failed
reloads, so the per-event failure handoff is no longer needed.
Tests that asserted on the synchronous reload now wire up the queue
(centralized in newTestS3ApiServerWithMemoryIAM) and poll via
waitForIdentity/waitForIdentityGone. Adds TestOnIamConfigChangeCoalescesBurstReloads
showing 50 events coalesce into <=3 reloads.
* fix(s3api/iam): skip non-JSON auxiliary files before failing IAM snapshot
Per review: the multi-file loaders unmarshal every entry in an IAM
directory, so a non-JSON auxiliary file (README, .DS_Store, a migration
backup such as identity.json.old) would hit the new empty/malformed
errors and reject the whole snapshot, blocking all later IAM reloads.
Only *.json files are IAM objects (SeaweedFS writes identities,
policies, service accounts and groups as <name>.json, and other call
sites already gate on the .json suffix). Skip non-.json entries at the
top of each loader loop, before reading content, so auxiliary files are
ignored while empty/malformed .json files still fail the snapshot.
Adds TestLoadConfigurationIgnoresNonJsonAuxiliaryFiles.
* fix(s3api/iam): reject IAM files with empty identifiers and skip aux in listing
Per review:
- ListPolicyNames listed every regular entry in the policies directory as a
policy name, including non-JSON auxiliary files, but GetPolicy cannot
retrieve them. Apply the same .json suffix filter used by the loader so
the list only exposes retrievable policies.
- json.Unmarshal accepts `{}` and unknown fields. The identity and group
loaders merge by the decoded Name (not the file name), so a `{}` file
could install an empty-key record and displace a real one; the
service-account loader accepted an empty Id. Validate Identity.Name,
Group.Name and ServiceAccount.Id (via validateServiceAccountId) after
unmarshal and fail the snapshot on empty identifiers.
Adds TestFilerEtcStoreListPolicyNamesSkipsNonJsonAuxiliary and
empty-identifier regression tests for identity, group and service-account
files.
* rust volume: honour is_last in the tail sender instead of rescanning
volume_tail_sender discarded the is_last flag from
binary_search_by_append_at_ns:
Ok((offset, _is_last)) => {
if offset.is_zero() { Ok(sb_size) } ...
is_last means the caller is already caught up. Go answers that with a
heartbeat and does not scan at all (volume_grpc_tail.go, `if isLastOne`).
Dropping it is expensive rather than untidy, because the branches interact:
when the search reports caught-up it returns Offset::default(), which is
zero, so the start offset falls back to sb_size -- the beginning of the
data -- and scan_raw_needles_from materialises every needle from there to
EOF into a Vec. The timestamp filter discards all of it, the loop sleeps
2s, and it happens again.
A volume being moved is marked read-only before the copy, so it is ALWAYS
caught up during the tail phase. Measured on one volume.move of a 2.15 GB
volume, sampling the source's cgroup anon every 2s against the move's own
phase output:
copying 16 -> 37 MB CopyFile streams correctly, stays bounded
tailing 904 -> 2166 -> 629 -> 2166 -> 342 -> 2173 -> 2179 MB
deleting 46 MB
Six full-volume allocate/free cycles in 35s, peak 2179 MB against a volume
of 2147 MiB. The destination never exceeded 35 MB, so this is entirely
source-side. Under a per-process memory cap it OOM-kills the source
whenever the volume exceeds the cap.
The ordering here is the whole fix and is easy to get wrong: resolve the
start offset and is_last under a brief lock, return the heartbeat
immediately when caught up, and only then reach the scan. An earlier cut
set the flag correctly but placed the early return after the block that
performs the scan -- the heartbeat fired and the destination received
nothing, yet every iteration still read the whole volume and discarded it.
Production showed no improvement (1770 MB across five cycles), which is
what caught it. The binary search is over the .idx and costs nothing; the
scan is the expensive part and must not run speculatively.
Three tests, and the last two matter as much as the first: a fix that
always reported "caught up" would make tailing silently lose needles, a
worse bug than the one being fixed. One asserts is_last for a caller at or
beyond the newest append_at_ns; one asserts NOT is_last for a caller that
is behind, so real tail data is still scanned and shipped; one asserts NOT
is_last when the only newer record is a delete, and that scanning from the
returned offset ships exactly that tombstone.
Left deliberately unfixed, and worth separate changes: the scan still
collects into a Vec rather than streaming through a visitor as Go's
ScanVolumeFileFrom does, and it runs while holding store.read(), the same
lock-across-a-large-read shape as #11235. Both are latent once the rescan
is gone, since remaining scans are bounded by genuinely new data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFr2v4BUqrXdgj4LEUAwVF
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3
* rust volume: resolve and scan the tail under one store guard
The tail sender took store.read() once for the binary search and again
for the scan. A vacuum commit takes the store write lock and swaps
.dat/.idx, so it could land between the two: the offset resolved against
the old files would then be applied to the new ones and start the scan
inside an unrelated record. The code before the is_last fix held a
single guard for both. Restore that, and scan only when the caller is not
caught up, so the caught-up heartbeat still skips the scan and is sent
outside the lock.
Also pin the compacted-volume boundary raised in review. Compaction
writes .idx in needle-id order in both Go and Rust, so the search can
report caught-up while an earlier row is newer; such a caller's since_ns
is the last row's timestamp, so those rows were in the files it copied.
A write made afterwards is appended as the final row, which the search
cannot step past. The new test asserts it still reaches the scan.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3
* rust volume: make the compaction tail test a genuine overwrite
The compaction regression test's second id=1 write reused the first
write's data, so write_needle's dedup short-circuit (is_file_unchanged)
returned without appending or updating append_at_ns. Compaction then
kept key 1's original (older) timestamp, so the test passed without
exercising the overwrite it describes -- key 2 was the final row only
because key 1 was never actually newer.
Give the overwrite distinct data so it appends a new record, and assert
key1_ns > key2_ns up front so a future dedup regression fails the test
instead of silently hollowing it out. Trim the verbose comments on the
tail sender and the binary-search tests to their essentials.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* Refactor parseDurationSeconds into a STSHandlers method
Convert the parseDurationSeconds wrapper from a package-level function
into a method on STSHandlers so it can reach the configured STS service.
No behavior change; the three AssumeRole* handlers now invoke it via
their receiver.
* Respect MaxSessionLength config in STS DurationSeconds validation
parseDurationSeconds validated DurationSeconds against a hardcoded
43200s (12h) ceiling, so raising maxSessionLength in iam.json above
12h had no effect on AssumeRole, AssumeRoleWithWebIdentity, or
AssumeRoleWithLDAPIdentity — requests were rejected at the handler
before reaching the service layer.
Derive the upper bound from the configured STS MaxSessionLength,
falling back to maxDurationSeconds (43200s) when unset. The service
layer (calculateSessionDuration) already caps the issued duration
at MaxSessionLength, so this only relaxes the input-validation gate.
* Add tests for STS DurationSeconds MaxSessionLength bound
Cover the configured MaxSessionLength upper bound, rejection above
it, fallback to the 43200s default when STS config is unset, the
900s minimum, and the empty-parameter nil path.
* Refactor validateSessionDurationSeconds into a STSService method
Convert validateSessionDurationSeconds from a package-level function
into a method on STSService so it can reach the configured STS config.
No behavior change; the three assume-role entry points in the service
(AssumeRoleForPrincipal, validateAssumeRoleWithWebIdentityRequest,
validateAssumeRoleWithCredentialsRequest) now invoke it via their
receiver.
* Respect MaxSessionLength config in STS service DurationSeconds validation
The STS service validateSessionDurationSeconds rejected DurationSeconds
above a hardcoded 43200s (12h) ceiling, so even after the handler
accepted a longer duration it was rejected again in the service layer
for AssumeRoleForPrincipal, AssumeRoleWithWebIdentity, and
AssumeRoleWithCredentials.
Derive the upper bound from the configured MaxSessionLength, falling
back to DefaultMaxSessionLength (43200s) when unset. The issued
duration is still capped at MaxSessionLength by calculateSessionDuration.
* Add tests for STS service DurationSeconds MaxSessionLength bound
Cover the configured MaxSessionLength upper bound, rejection above
it, fallback to the 43200s default when STS config is unset, the
900s minimum, and the nil DurationSeconds path.
* Preserve capping when MaxSessionLength is below the API minimum
Deriving the DurationSeconds upper bound directly from MaxSessionLength
created an empty valid range when MaxSessionLength is configured below
the 900s API minimum, rejecting every explicit DurationSeconds that the
old code silently capped via calculateSessionDuration.
Only apply the configured MaxSessionLength as the upper bound when it is
at least minDurationSeconds; otherwise keep the default bound and let
calculateSessionDuration enforce the shorter configured limit.
* Add tests for sub-minimum MaxSessionLength capping behavior
Verify that a MaxSessionLength below the 900s API minimum keeps the
default upper bound so explicit DurationSeconds within the default
range are still accepted (and later capped by calculateSessionDuration).
* rust volume: stop glibc retaining freed EC buffers as unreturnable heap
A Rust volume server doing EC work accumulates hundreds of MB of resident
anonymous memory that it never gives back, and under a hard cgroup
MemoryMax that ends in an OOM kill while most of the resident set is
free-but-unreturned.
It is not a leak. glibc serves allocations >= M_MMAP_THRESHOLD with mmap
and munmaps them on free, but the threshold is ADAPTIVE: freeing an
mmap'd block raises it toward that block's size, up to 32 MiB. EC
reconstruction and needle reassembly allocate large short-lived buffers,
so the first few train the threshold upward and every later buffer is
carved from the heap instead. Heap pages only return to the OS from the
top of the arena, so they stay resident for the life of the process --
reusable, but anonymous, and anonymous pages cannot be reclaimed under
pressure the way page cache can. The retained footprint is exactly the
headroom a burst of maintenance work needs.
Measured on a 17-node cluster (EC 10+4, --index=redb), one node, two
identical `ec.scrub -mode full` rounds over 10912 EC files each, same
unit restarted with and without a pinned threshold:
baseline round 1 round 2 60s idle
default (adaptive) 10 MB 84 MB 88 MB 88 MB
pinned threshold 10 MB 13 MB 14 MB 14 MB
78 MB retained versus 4 MB for identical work. On heavier mixed scrub
workloads the same effect reached ~600 MB per volume server against a
3 GiB cap, and restarting the process was the only way to release it.
Calling mallopt(M_MMAP_THRESHOLD, ...) sets the threshold and disables
the dynamic adjustment. Pin it to glibc's own default rather than
inventing a value: the goal is to stop the adaptation, not to second-guess
the default. MALLOC_MMAP_THRESHOLD_ still wins if an operator sets it,
glibc-only, and a failed mallopt is logged rather than fatal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFr2v4BUqrXdgj4LEUAwVF
* Address PR review: validate env overrides, honour GLIBC_TUNABLES, fix non-glibc test compile
Three review-bot findings on seaweed-volume/src/malloc_tuning.rs:
1. (CodeRabbit) The test used cfg!(...), which keeps both branches in
compilation. On non-glibc targets DEFAULT_MMAP_THRESHOLD is undefined,
so the test failed to compile. Split into #[cfg]-gated tests so each
branch only references items defined for that target.
2. (Greptile) MALLOC_MMAP_THRESHOLD_ was checked by presence only. An
empty or non-numeric value makes glibc ignore the override while we still
skipped mallopt, leaving the adaptive threshold enabled -- exactly the
behaviour this module exists to prevent. Now we defer only when the value
is non-empty and parses as an integer; otherwise we fall through to
pinning.
3. (Codex) The modern GLIBC_TUNABLES=glibc.malloc.mmap_threshold=... tunable
was missed, so mallopt could overwrite an operator's explicit tunable. Now
we detect that tunable (with the same validation) and defer to it.
The override check moved into the glibc-gated inner function, so off glibc
pin_mmap_threshold() always reports NotApplicable regardless of any
allocator env vars that happen to be set. The startup log for DeferredToEnv
is reworded to cover both override sources. Added tests for the override
parsers and the off-glibc no-op.
* Address round-2 review: match glibc's actual override parsing
Three follow-up review-bot findings after the first round of fixes, all
rooted in our validation not matching how glibc actually parses the
overrides:
1. (Greptile, P1) parse::<i64>() accepted negative values like "-1" and
returned DeferredToEnv, but glibc's threshold is unsigned and rejects
negatives — so we skipped mallopt while glibc also ignored the override,
leaving the adaptive threshold enabled. Now we reject negatives and
zero.
2. (Devin, BUG) glibc parses thresholds as unsigned (strtoul for tunables,
atoi for the legacy var). Values above i64::MAX are valid for glibc but
were rejected by parse::<i64>(), so we pinned 128 KiB over the operator's
explicit setting. Now we parse as u64, accepting the full unsigned range.
3. (CodeRabbit, Major) Two issues in usable_glibc_tunable_threshold:
a. A malformed sibling entry (e.g. glibc.malloc.check=2=2:...) makes
glibc reject the entire GLIBC_TUNABLES string, but our per-entry scan
still returned true for the valid-looking mmap_threshold entry. Now
we validate every entry (exactly one '=') before accepting any.
b. Hex values (0x20000) are accepted by glibc's strtoul but were rejected
by parse::<i64>(). Now parse_strtoul_threshold handles 0x-prefixed hex.
MALLOC_MMAP_THRESHOLD_ stays decimal-only (atoi), matching glibc.
Added regression tests for negatives, zero, >i64::MAX, hex tunables, and
malformed mixed GLIBC_TUNABLES entries. Verified: clippy clean and tests
pass on macOS (non-glibc); glibc-gated code type-checks for
x86_64-unknown-linux-gnu.
* Address round-3 review: match glibc's actual override parsing
Three follow-up review-bot findings (Greptile P1, Devin BUG, CodeRabbit
Major) all on the same issue: the round-2 fix rejected negative and zero
override values, but glibc actually accepts them.
Verified against the glibc source (malloc/malloc.c, malloc/arena.c,
elf/dl-tunables.c, elf/dl-misc.c):
- do_set_mmap_threshold(size_t value) does NO clamping — it just sets
mp_.mmap_threshold = value and mp_.no_dyn_threshold = 1.
- MALLOC_MMAP_THRESHOLD_: glibc calls atoi(value) then mallopt, which
always sets the threshold and disables dynamic adjustment — even for
empty, negative, or non-numeric values (atoi returns 0). So ANY
presence of the variable means the operator's override is in effect.
Reverted to presence-only check for the legacy variable. The round-1
Greptile comment claiming glibc "cannot apply the override" for
empty/malformed values was incorrect.
- GLIBC_TUNABLES: glibc parses values with _dl_strtoul (elf/dl-misc.c),
which accepts decimal, 0x hex, 0 octal, an optional sign (negatives
wrap to unsigned long), and requires the entire value consumed
(tunable_parse_num checks endptr == strval + len). Replaced
parse_strtoul_threshold with dl_strtoul_consumes_all that replicates
_dl_strtoul's parsing and checks full consumption. Now accepts -1
(wraps to SIZE_MAX), 0, 0x20000, 010 (octal), and values above
i64::MAX.
The duplicate-= validation for GLIBC_TUNABLES (from round 1) is kept —
glibc's parse_tunables_string returns -1 if any entry's value contains
a duplicate =, rejecting the entire string.
Added dl_strtoul_consumes_all tests covering decimal, hex, octal,
negative, zero, empty, whitespace, trailing garbage, and sign-only
inputs. Updated usable_glibc_tunable_threshold tests to accept
negative, zero, and empty values. Verified: clippy clean and tests
pass on macOS (non-glibc); glibc-gated code type-checks and clippy
clean for x86_64-unknown-linux-gnu.
* Address round-4 review: add overflow detection, fix sign-only test assertions
Two Greptile P1 findings:
1. Overflowing tunables bypass threshold pinning: dl_strtoul_consumes_all
consumed every digit and returned true for values like
18446744073709551616 (u64::MAX + 1), but glibc's _dl_strtoul stops at
the overflowing digit (sets endptr there, returns UINT64_MAX), so
tunable_parse_num rejects the value (endptr != strval + len). Added
overflow detection matching glibc's cutoff/cutlim logic — on overflow,
the parser stops and returns false.
2. Sign-only parser assertions fail: the test asserted
!dl_strtoul_consumes_all("-") and !dl_strtoul_consumes_all("+"), but
_dl_strtoul skips the sign, finds no digit, sets endptr to the position
after the sign (== end of string), and returns 0. tunable_parse_num
sees endptr == strval + len → true. So glibc accepts sign-only strings
as value 0. Fixed the test assertions to expect true.
Also fixed "0x" with no hex digits: _dl_strtoul parses "0" as octal, then
stops at "x" (not an octal digit), so endptr != end of string → rejected.
The base-detection now requires a hex digit after "0x" before switching
to hex; otherwise "0" is parsed as octal and "x" stops the parser.
Added overflow regression tests: 18446744073709551616 (u64::MAX + 1),
99999999999999999999 (20 nines), 0x10000000000000000 (2^64). Verified:
clippy clean and tests pass on macOS (non-glibc); glibc-gated code
type-checks and clippy clean for x86_64-unknown-linux-gnu.
* Address round-5 review: accept bare 0x prefix, remove unused helper
Two review-bot findings (Devin BUG + CodeRabbit Major) on the same issue:
the round-4 fix required a hex digit after "0x" before switching to hex
base, but glibc's _dl_strtoul unconditionally advances past "0x"/"0X"
when the first char is '0' and the next is 'x'/'X' — even if no hex digit
follows. In that case the digit loop breaks immediately, endptr reaches
the end, and the value is 0. tunable_parse_num accepts it.
Removed the is_digit_in_base lookahead from the base-detection condition
and the now-unused is_digit_in_base helper. Updated the test assertions
for "0x" and "0X" to expect true (accepted as value 0).
The Greptile P1 overflow comment is invalid: glibc's _dl_strtoul rejects
18446744073709551616 (u64::MAX + 1) — on overflow it sets endptr to the
overflowing digit (not end of string) and returns UINT64_MAX, so
tunable_parse_num sees endptr != strval + len and rejects. My
implementation correctly returns false for this value, matching glibc.
Verified: clippy clean and tests pass on macOS (non-glibc); glibc-gated
code type-checks and clippy clean for x86_64-unknown-linux-gnu.
* Address round-6 review: rewrite tunable parser to match glibc exactly
Two Greptile P1 comments (3975151906, 3975151911) both invalid, but
investigation revealed a real bug in the split(':')-based parser:
Bug: usable_glibc_tunable_threshold used split(':') which loses the
distinction between an entry terminated by ':' (glibc skips it) and one
terminated by '\0' with no '=' (glibc rejects the entire string). Examples:
- "glibc.malloc.mmap_threshold=262144:glibc.cpu.x" (no '=' at end):
glibc rejects entire string, old code accepted it.
- "glibc.malloc.mmap_threshold=262144:" (trailing ':'):
glibc rejects entire string, old code accepted it.
Fix: replaced split(':') with a character-by-character parser matching
glibc's parse_tunables_string exactly. The parser tracks position in the
original string and correctly handles all three terminators ('=', ':', '\0')
for both name and value scanning.
Comment 3975151906 (near-maximum values): Invalid. Verified against
_dl_strtoul: for 18446744073709551615 (u64::MAX), cutoff = u64::MAX/10,
cutlim = u64::MAX%10 = 5. After 19 digits result == cutoff. 20th digit 5:
overflow check (digval > cutlim) is 5 > 5 = false → no overflow. glibc
accepts u64::MAX. Added regression test asserting it's accepted.
Comment 3975151911 (later malformed entry): Invalid. Verified against
parse_tunables (elf/dl-tunables.c): when parse_tunables_string returns -1,
parse_tunables prints a warning and returns immediately without applying
ANY tunable — including ones already parsed into the array. Added
regression test for "threshold=262144:check=2=2" (threshold before
malformed sibling) asserting it's rejected.
Added regression tests: u64::MAX accepted, threshold-before-malformed
rejected, no-'=' at end rejected, trailing ':' rejected, leading ':'
accepted. Verified: clippy clean and tests pass on macOS; glibc-gated
code type-checks and clippy clean for x86_64-unknown-linux-gnu.
* Fix CI: correct hex trailing-garbage test assertion
The test asserted !dl_strtoul_consumes_all("0x20000abc"), but in hex
mode a-f are valid digits — "0x20000abc" is a valid hex number
(0x20000abc = 536874044), not trailing garbage. _dl_strtoul consumes
the entire string and tunable_parse_num accepts it. The assertion
failed on Linux CI where the glibc-gated test actually runs.
Replaced with "0x20000g" — 'g' is not a hex digit, so _dl_strtoul
stops at 'g' and tunable_parse_num rejects the value.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(volume): harden volume copy validation
* fix(volume): use stream context for ReadVolumeFileStatus in VolumeCopy
ReadVolumeFileStatus ran on context.Background() while the adjacent
VolumeStatus call used stream.Context(), an inconsistency left over
from the context revert in #11252. Use stream.Context() consistently
so the source status check is cancelled with the VolumeCopy stream.
* fix(volume): reserve destination before deleting existing replica
FindFreeLocation now runs before DeleteVolume so a full target fails
without destroying the existing replica. Previously, when the initial
VolumeStatus check failed (advisory) but ReadVolumeFileStatus
succeeded, the existing replica was deleted before a destination was
reserved, risking data loss if no location had enough free space.
Add a regression test verifying the existing replica survives when
the destination is full and the initial status check fails.
* fix(volume): count replaced replica slot in FindFreeLocation
FindFreeLocation now accepts the volume being replaced so its slot is
treated as available. Without this, a location at its MaxVolumeCount
limit could not replace its sole replica even though deleting it would
free the slot. VolumeCopy passes the volume ID so destination selection
succeeds before the existing replica is deleted.
Add TestVolumeCopyReplacesReplicaAtSlotLimit covering a single-slot
location that must replace its only replica.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: end local-only metadata subscriptions when remote peers appear
SubscribeMetadata delegates to SubscribeLocalMetadata whenever the
MetaAggregator knows no remote peers at stream setup. Peer discovery is
asynchronous with the gRPC server accepting streams: the master announces
filers after Filer.Init, via ListExistingPeerUpdates and OnPeerUpdate.
A subscriber that connects inside that window is pinned to a filer-local
stream for its whole life, silently missing every other filer's writes.
For filer.remote.sync in a multi-filer cluster this means the remote tier
permanently stops receiving writes served by other filers (#11247).
End the delegated local stream when the first remote peer appears, so the
client reconnects into the aggregated stream. The end surfaces as an
error, not a clean EOF: RetryUntil-driven followers (mount meta cache,
s3api IAM) treat a clean end as following finished and stop
reconnecting. The arrival channel is armed under the same lock as the
peer check in RemotePeerArrivedChan, so a peer learned in between sends
the stream straight to the aggregated path instead of parking on a
channel that would never fire.
A standalone filer is unaffected: no peer ever appears, the channel
never fires, and the local stream serves indefinitely.
Fixes#11247
* filer: interrupt disk replay on peer arrival, trim comments
Check upgradeOnRemotePeer inside eachLogEntryFn and chunkDiskPass so a
peer arriving during a backlog replay stops the stream before the
cursor advances past older remote events. Wrap errAggregationUpgrade
with StopReadingError so LoopProcessLogData does not log it. Remove
issue references from comments and trim verbose commentary.
* filer: check upgrade signal between ref batches
Pass upgradeOnRemotePeer to sendRefsBatched so a peer arriving while
refs are shipped to a slow client is detected between batches, not
only after the full batch completes.
* filer: interrupt gap park on peer arrival
Pass upgradeOnRemotePeer through gapPass to parkOnGap so a peer
arriving during a gap park ends the stream immediately instead of
waiting for the retry timer (up to one minute).
---------
Co-authored-by: Tyagiquamar <Tyagiquamar@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: add isActionExplicitlyDeniedByApplicablePolicies helper
Add a helper that reports whether any applicable identity-side policy
(attached IAM policies, enabled-group policies, or the IAM-integration
session policy) explicitly denies an action. It reuses the existing
evaluateAttachedIAMPolicies, resolveS3AuthTarget, buildPrincipalARN, and
isActionExplicitlyDeniedByIAM helpers, and fails closed on evaluation
errors. A nil identity has no identity-side policy plane, so the helper
returns false to keep the bucket policy authoritative for anonymous
access. No behavior change yet; the next commits apply it to the two
bucket-policy Allow short-circuits.
* s3: enforce identity explicit Deny before bucket-policy Allow
authRequestWithAuthType short-circuits on a matching bucket-policy Allow
and skips VerifyActionPermission, so an explicit Deny in an authenticated
identity attached, group, or session policy is bypassed. A non-admin
principal with s3:PutBucketPolicy can install a bucket-policy Allow for
itself and read an object its identity policy explicitly denies.
Before honoring a bucket-policy Allow, check the applicable identity-side
policies for a matching explicit Deny via the new
isActionExplicitlyDeniedByApplicablePolicies helper, and fail closed.
The cross-account behavior is preserved: a bucket Allow still supplies the
Allow an identity policy omits (implicit denial), and a nil identity keeps
the bucket policy authoritative for anonymous access.
Regression tests cover the explicit-Deny override, the implicit-deny
Allow preservation, and the unmatched-key fall-through control.
* s3: enforce identity explicit Deny in secondary object-key auth
authorizeObjectKeyAction authorizes keys the request URL does not name
(CopySource, DeleteObjects body keys, POST Object form keys) and shares
the same bucket-policy Allow short-circuit as the primary path, so an
explicit Deny in the identity, group, or session policy is bypassed the
same way when a bucket policy allows the secondary key.
Apply isActionExplicitlyDeniedByApplicablePolicies before accepting the
bucket-policy Allow, mirroring the primary path. A regression test covers
AuthorizeCopySource for both the explicit-Deny override and the
implicit-deny Allow preservation.
* rust: prevent phantom volumes + validate collection hint in mount_volume_by_id
The collection-hint path (and the find_volume_file_base fallback) called
create_volume on any matching .vif/.idx sidecar. create_volume ->
Volume::new -> load(create_dat_if_missing=true) writes an empty .dat and
registers a phantom normal volume, which can shadow a real EC volume whose
.ecx lives on a sibling disk. This reintroduces the phantom-volume bug the
codebase explicitly guards against in load_existing_volumes.
Apply the same guard load_existing_volumes uses to both paths: only mount
when a real .dat is present or the .vif references a remote-tiered file;
otherwise skip the candidate (no phantom). Also reject path-bearing
collection hints ('/', '\\', '..') so the shortcut cannot route .dat
creation outside the storage directory, falling back to the safe scan.
Adds 3 regression tests; all 336 storage:: tests pass.
Addresses Devin + Greptile review comments on PR #11249.
* rust: address review — .note guard, multi-candidate scan, foo..bar hint
Address the four review comments on #11254:
1. Greptile (P1): contains("..") rejected valid collections like "foo..bar".
Replaced with collection != ".." — volume_file_name joins with "_" so a
".." inside a name is part of the filename, not a parent reference. Only
the exact ".." name is rejected. Added a test that "foo..bar" mounts.
2. Devin #0001 (bug): mount_volume_by_id did not check the .note marker, so
an interrupted VolumeCopy could mount as a live (truncated) volume. Added
a .note check before create_volume in both the collection-hint path and
the fallback — a candidate with .note is skipped (matches
load_existing_volumes). Added a test covering both paths.
3. Devin #0002 + CodeRabbit (major): find_volume_file_base returned only the
first matching candidate, so a lone sidecar on disk 0 hid a real .dat on
disk 1 (the split-disk EC layout the phantom guard protects against).
Added find_volume_file_bases (plural) that collects all candidates; the
fallback now iterates every candidate and mounts the first with a real
.dat or remote .vif. find_volume_file_base delegates to it for
configure_volume. Added a two-disk test: sidecar on disk 0, real .dat on
disk 1 — mount succeeds from disk 1.
All 339 storage:: tests pass (6 mount_volume_by_id tests).
* rust: continue past create_volume failure in mount_volume_by_id
Address Devin review comment on #11254: when create_volume fails on an
earlier candidate (e.g. an unreadable .dat), mount_volume_by_id returned
the error immediately instead of trying later candidates. A valid volume
on another disk remained unmounted.
Both the collection-hint loop and the find_volume_file_bases fallback now
remember the last error and continue scanning. A successful mount returns
immediately; if no candidate succeeds, the last error (or NotFound) is
returned. Matches DiskLocation::open_volumes and Go Store.mountVolume.
Added test_mount_volume_by_id_continues_past_open_failure (chmod 000 .dat
on disk 0, real volume on disk 1, mounts from disk 1).
All 340 storage:: tests pass.
* iceberg: return 401 for invalid or expired Bearer tokens
BUG-0001: when the OAuth JWT expired, Server.Auth fell through to the S3
SigV4 authenticator, which rejects the "Authorization: Bearer" scheme
with NotImplemented — a 501. Iceberg clients (Java OAuth2Manager,
pyiceberg) only refresh tokens on 401, so they retried the dead token
forever: RisingWave sinks stalled and Doris catalog queries failed every
token TTL (1h) until the client process was restarted.
A request carrying a Bearer header is an Iceberg REST client: answer 401
(+ WWW-Authenticate: Bearer, RFC 6750) when the token fails, and only
fall through to the S3 authenticator when no Bearer header is present.
* iceberg: make OAuth token TTL configurable via ICEBERG_OAUTH_TOKEN_EXPIRY
BUG-0001 follow-up: production evidence shows Iceberg Java 1.10.x
clients (RisingWave connector node, Doris FE) never re-fetch tokens on
401 — the sink stalled again on token expiry even with the 501→401 fix,
and no POST /v1/oauth/tokens appeared in server logs across dozens of
retries. 401 is necessary but not sufficient for these clients.
The TTL was hardcoded to 3600 with no knob. Read the expiry (seconds)
from ICEBERG_OAUTH_TOKEN_EXPIRY, defaulting to 3600, so deployments can
issue longer-lived tokens (e.g. 86400) to survive client restart cycles.
* iceberg: support OAuth token exchange (RFC 8693) for client refresh
Decompiling the Iceberg Java 1.10.1 client bundled with Doris FE showed
the missing half of BUG-0001: OAuth2Manager refreshes via token-exchange
(AuthConfig.exchangeEnabled defaults to true — the client_credentials
re-fetch branch only runs with exchange disabled), so a server that only
accepts client_credentials leaves Iceberg clients unable to ever refresh
their token, regardless of 401 correctness.
Accept grant_type=urn:ietf:params:oauth:grant-type:token-exchange on
POST /v1/oauth/tokens: verify the subject_token signature against the
issuing credential, allow exchange within a recovery grace window
(max(2*TTL, 1h), capped 24h) so clients holding tokens that expired
while the grant was unsupported recover without a restart, and mint a
fresh access token with the configured TTL.
* iceberg: harden OAuth token exchange and Bearer matching per review
- match the Bearer scheme case-insensitively (RFC 7235), like
authenticateBearer already does
- accept optional client authentication on the token-exchange grant
(Basic or form credentials, bound to the subject token's client);
expired subject tokens now require it. Iceberg Java's proactive
refresh sends Bearer-only headers, so the grant cannot require it
- reject subject tokens without an exp claim, and re-check the issuer
on the verified claims
- unauthenticated exchange cannot extend the lifetime past the
subject token's own expiry (no chain-refresh from a leaked token)
- return 400 invalid_grant per RFC 6749 §5.2 (was 401)
- include issued_token_type on exchange responses (RFC 8693)
- clamp ICEBERG_OAUTH_TOKEN_EXPIRY to 365d so Duration math cannot
overflow into already-expired tokens
* iceberg: give authenticated token exchanges a fresh full TTL
The remaining-lifetime cap only guards unauthenticated (Bearer-only)
exchanges; an authenticated client renewing a live token must get the
full configured TTL, matching client_credentials.
* iceberg: reject token exchange when no lifetime remains
A Bearer-only exchange with under a second of subject lifetime would
mint a token with expires_in: 0. Reject with invalid_grant instead.
* iceberg: pin near-expiry test token to the next second boundary
jwt/v5 serializes exp at one-second precision, so a 300 ms offset can
round into the current second and route the test through the expired
branch instead of the ttlSeconds<=0 guard. Mint the subject with the
next whole-second expiry: live at exchange time, deterministically
under a second of remaining lifetime.
* iceberg: drop internal ticket reference from comments
* iceberg: clamp oversized OAuth TTLs on 32-bit platforms
strconv.Atoi on an int-sized value fails with ErrRange on 386, so an
oversized ICEBERG_OAUTH_TOKEN_EXPIRY silently fell back to the default
instead of clamping. Parse in 64-bit space and clamp, then narrow.
* iceberg: make OAuth TTL narrowing explicit
* iceberg: disable legacy OAuth in PyIceberg integration tests
* rust volume: move the crate to edition 2024
Edition 2024 turns three things in this crate into hard errors, and changes
drop order in a further 34 places without changing compilation. The compiler
errors are fixed here; the silent changes were audited against
`RUSTFLAGS='-W rust-2024-compatibility' cargo check --all-targets` output
captured before the flip, since edition 2024 stops reporting them.
`std::env::set_var`/`remove_var` are unsafe as of 2024 because they race with
concurrent readers. All six call sites are safe by construction rather than by
assertion, and the SAFETY comments say why: the build script runs
single-threaded before anything else in the process, and every test reaching
the `config.rs` helpers holds `process_state_lock()` for the duration.
The two `ref` bindings in handlers.rs sit in patterns that already borrow
implicitly, so removing the modifier leaves both bindings at `&String`.
On the 34 drop-order sites: no lock guard's scope is extended anywhere, and
`volume.rs` has none. Most are moved-from `Option`/`Result` husks — `if let
Some(v) = map.remove(&k)`, `while let Some(m) = stream.next().await` — where
the value is moved into the binding and the temporary has nothing left to drop;
where closing order actually matters these paths already call `v.close()`,
`ec_vol.destroy()` or `drop(writer)` explicitly. Two sites get strictly better
ordering: the metrics read guard in `run_metrics_push_loop` shrinks to the end
of its initializer block (it never crossed an `.await` either way), and an EC
test now closes the volume's descriptors before the `TempDir` removes the
directory.
No `rust-version` is declared. Edition 2024 needs rustc 1.85, but that is not
the binding constraint — the dependency tree already requires 1.91.1 through
the `aws-sdk-s3`/`aws-smithy-*` family, so `cargo +1.85 check` fails on the
deps regardless. CI builds on `dtolnay/rust-toolchain@stable`.
`vendor/reed-solomon-erasure` is a separate package and keeps edition 2021.
Cargo.lock is unchanged despite edition 2024 implying resolver 3.
Verified: `cargo test` 551 passed / 0 failed, `cargo test
--no-default-features` 550 passed / 0 failed (the two feature sets produce an
identical migration site list), `cargo build --release` clean. No automated
test covers shutdown ordering, so the channel and runtime sites in `main.rs`,
`write_queue.rs` and `grpc_server.rs` were read individually.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC
* rust volume: address edition-2024 review feedback
Three fixes from review of the edition bump.
Serialize the two environment-reading tests. The SAFETY comments on the
`env::set_var`/`remove_var` helpers claim every test touching the environment
holds `process_state_lock()`, but `test_resolve_config_defaults_dir_to_platform_temp_dir`
and `test_resolve_config_index_accepts_redb_and_leveldb_aliases` called
`resolve_config` — which reads HOME/USERPROFILE, SEAWEED_WRITE_QUEUE and the
WEED_* set — without taking it. `set_var` is unsafe precisely because a
concurrent *reader* is UB, not only a concurrent writer, so the comment was
overclaiming. An audit of the module found exactly these two; every other
environment-touching test already held the lock. The race predates edition
2024, which only made the requirement explicit.
Declare `rust-version = "1.91.1"`. The edition needs 1.85, but that was never
the binding constraint: `cargo +1.90 check --all-targets` fails on the
`aws-sdk-s3`/`aws-smithy-*` family, and 1.91.1 checks clean. Declaring the
verified floor turns a wall of per-dependency errors into one clear message.
Cargo.lock is unchanged despite this making the resolver MSRV-aware.
Update the README, which advertised "Rust 1.75+ (2021 edition)". 1.75 was
already stale before this branch — the tree has needed 1.91 for a while.
Verified: `cargo test` 551 passed / 0 failed, `cargo test --no-default-features`
550 passed / 0 failed, `cargo build --release` clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC
* rust volume: state the exact MSRV patch release in the README
The README said "Rust 1.91+", which reads as 1.91.0 and is wrong by one patch
release: `cargo +1.91.0 check --all-targets` fails on the aws-sdk-s3 family,
`cargo +1.91.1` passes. Say 1.91.1+, matching `rust-version` in Cargo.toml, and
call out that the patch component is load-bearing so nobody installs 1.91.0 and
hits the same wall.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC
* rust worker: move the workspace to edition 2024
Moves the seaweed-worker workspace (core, lance, sort) from edition 2021 to
2024, the same migration seaweed-volume just got in this branch.
Edition 2024 turns exactly one thing in this workspace into a hard error. The
baseline came from RUSTFLAGS='-W rust-2024-compatibility' cargo check
--all-targets, run before the flip; unlike seaweed-volume's 34 silent +
8 hard sites, the worker reports only the one hard site and no
tail_expr_drop_order or if_let_rescope sites at all. The worker is a much
smaller crate and none of its expressions hold a guard or temporary whose drop
order the edition changes, so there is nothing to audit on the silent side.
Fixed (1 site):
std::env::set_var is unsafe as of 2024 because it races with concurrent
readers. The single call is in crates/core/build.rs, which sets PROTOC from
protoc_bin_vendored the way seaweed-volume's build script does. A build
script's main runs single-threaded before anything else in the process, so
no other thread can be reading the environment concurrently; the SAFETY comment
says so. There are no config.rs-style test helpers here -- the worker's tests
do not mutate the environment -- so unlike the volume crate there are no
process_state_lock() callers to audit.
No redundant ref bindings to clean up: a grep for ref across the three
crates finds none.
MSRV:
rust-version = "1.94.1", verified rather than inferred. Edition 2024 only
needs 1.85, but the dependency tree needs more: lance's aws feature pulls in
a newer cut of the same aws-sdk-*/aws-smithy-* family that sets
seaweed-volume's 1.91.1 floor, and that newer cut requires 1.94.1.
cargo +1.94.0 check --all-targets fails on that family; cargo +1.94.1
check --all-targets is clean. The worker's floor is therefore higher than
the volume's, and moves with lance and the AWS SDK rather than with the
edition. CI builds on dtolnay/rust-toolchain@stable, so nothing changes
there.
The edition is set once in [workspace.package] and inherited by each member
via edition.workspace = true; rust-version is added the same way. The
workspace keeps its explicit resolver = "2" -- edition 2024 would default to
resolver 3, but the pin is deliberate and Cargo.lock is unchanged by this
commit either way.
The README gains a "Requires Rust 1.94.1+ (2024 edition)" line in its Building
section, matching the one seaweed-volume's README now carries, and calling out
that the patch release is load-bearing (1.94.0 does not build) so nobody
installs 1.94.0 and hits the same wall.
Verification:
* cargo check --all-targets -- clean, zero warnings (default toolchain 1.97)
* cargo +1.94.1 check --all-targets -- clean
* cargo +1.94.0 check --all-targets -- fails on the AWS SDK, as claimed
* cargo test --all-targets -- 40 passed, 0 failed
(core 13, sort 11, lance lib 3, lance bin 2, compaction 6, lifecycle 1,
sort integration 4)
* Cargo.lock unchanged
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
FullPath.DirAndName() and FullPath.Name() used filepath.Split, which is
OS-dependent: on Windows it treats backslash as a path separator,
corrupting filer paths that contain literal backslashes. Filer paths
always use "/" as the separator, so switch to path.Split and path.Join
which only split on "/" regardless of the host OS.
This fixes the backslash case from #11243 where a file saved as
/test/special\reverseslash4.jpg was stored with a corrupted path on
Windows filer builds. The #, ?, and % cases from the same issue are
client-side URL-encoding problems (the server never receives the raw
characters), but once the client properly percent-encodes them the
server now handles the decoded path correctly on all platforms.
* fix: rebase on latest master before pushing star history chart
The daily star history workflow git push was rejected with a
non-fast-forward error because new commits landed on master between
the checkout and the push. Fetch full history (fetch-depth: 0) and
rebase the generated commit on top of the latest remote branch before
pushing so the workflow no longer fails when master has moved.
* fix: retry rebase-and-push to handle concurrent master updates
Address review feedback: a one-shot rebase still races if master advances
between the rebase and the push. Match the bounded retry loop used by
java_release.yml — push first, and on rejection rebase and retry up to
five times before failing.
* fix: serialize runs and ensure every rebase is followed by a push
Address review feedback:
- Devin (line 54-55): the old loop rebased after the 5th failed push but
never pushed the rebased commit. Restructure so each rebase (attempts
2-5) is followed by a push attempt, with a clear 5-attempt cap.
- Greptile: overlapping runs could conflict on the SVG during rebase.
Add a concurrency group (cancel-in-progress: true, matching the repo
convention) so only one chart regeneration runs at a time.
* fix: scope concurrency by ref and guard rebase against transient failures
Address review feedback:
- Devin (line 16): the global star-history concurrency group let a manual
run on another branch cancel an in-flight daily master update. Scope
the group by github.ref so only same-branch runs cancel each other.
- Greptile (line 58): git pull --rebase runs under the fail-fast shell,
so a transient fetch error or conflict aborted the whole step before
remaining attempts ran. Guard the rebase so a failure aborts the
in-progress rebase and continues to the next attempt instead.
* fix(volume): track EC shard read errors and unmount on faulty media
Extract the volume EIO tracker into a reusable IoErrorTracker and add the
same tracking to EcVolume. Sustained EIO on .ecx lookups or .ecd shard
reads now unmounts the EC volume in the heartbeat (without deleting
files) so the master re-replicates from healthy peers, mirroring the
existing volume replica quarantine.
Closes#11227 (EC shard unmount).
* rust(volume): mirror EC shard read error tracking and unmount
Add EIO tracking to the Rust EcVolume mirroring Go: a streak counter
with IO_ERROR_TOLERANCE, a sticky quarantine flag, and unmount (not
file deletion) in the heartbeat so the master re-replicates from
healthy peers.
* feat(metrics): expose storage IO error counter and quarantine gauge
Add a storage_io_error_total counter incremented on every EIO recorded
by the volume or EC shard tracker, and an io_quarantine gauge labelled
by kind (volume/ec_shard) reflecting the count of replicas suppressed
in the heartbeat. Mirrored in Go and Rust.
* feat(healthz): report 503 when local replicas are IO-quarantined
Add Store.HasIoQuarantine (Go) / Store::has_io_quarantine (Rust) and
have /healthz return 503 when any local volume or EC shard is
quarantined due to sustained storage-media EIO, so a load balancer
can drain a server whose underlying media is faulty. Mirrored in Go
and Rust.
* fix(volume): keep quarantined EC volumes in memory and reset EIO on success
Address review feedback: instead of unloading quarantined EC volumes
(which discards the quarantine state healthz needs), keep them in
memory and just skip them from heartbeat reporting, mirroring the
regular volume quarantine. Also clear the EIO streak on successful
.ecx reads in Rust so a transient error does not accumulate, and add
an ec_shard label to the io_quarantine gauge in both Go and Rust.
* fix(volume): exclude quarantined EC shards from heartbeat and add Rust volume tolerance
Address review feedback:
- Filter quarantined EC volumes from CollectErasureCodingHeartbeat
(Go) and collect_ec_shard_delta_messages / collect_live_ec_shards
(Rust) so the master stops advertising faulty shards and
re-replicates from healthy peers.
- Add consecutive EIO count and sticky quarantine to the Rust
regular Volume, mirroring Go IoErrorTracker: a single EIO no
longer deletes the replica; the heartbeat quarantines after the
tolerance threshold and keeps the volume in memory.
- Use the quarantine flag (not last_io_error) in has_io_quarantine
so /healthz reflects sustained, not transient, failures.
* fix(volume): make Rust quarantined volumes read-only and wire recovery
Address Devin review:
- Set no_write_or_delete on Rust volumes when quarantined in the
heartbeat, so cached or direct clients cannot mutate a faulty
replica after the master removes it (mirrors Go).
- Wire reset_io_error_state into Volume::set_writable so an operator
making a volume writable again clears the sticky quarantine and
the volume re-enters heartbeat rotation.
* fix(volume): clear EC quarantine on shard re-mount for operator recovery
Address Greptile review: re-mounting EC shards (Go loadEcShardWithIdxDir
/ Rust mount_ec_shards_with_idx_dir) now calls ResetIoErrorState on the
existing EcVolume, giving operators a documented recovery path that
clears the sticky quarantine and returns the EC volume to heartbeat
rotation. Mirrored in Go and Rust.
* fix(volume): do not clear EC quarantine on routine shard mounts
Address review feedback: clearing the EC IO quarantine on every mount
(including duplicate, retry, sibling-shard, and reconciliation mounts)
is too aggressive and can re-advertise known-bad shards before the
storage media has been validated. Remove the automatic reset from the
mount path; quarantine clears naturally on restart or full unmount
when a fresh EcVolume is created with clean state.
* test(volume): update Rust IO error test for quarantine semantics
The heartbeat now quarantines a volume with sustained EIO (keeps it
mounted, makes it read-only, omits it from heartbeat) instead of
deleting it. Update test_collect_heartbeat_deletes_io_error_volume to
assert the volume stays in the store with no_write_or_delete set, and
update set_last_io_error_for_test to set the consecutive error count
at the tolerance threshold so the test reflects a sustained error.
* fix(volume): reset EIO streak after full write and match Windows media errors
Move the success-side EIO reset from append_needle (after write_all only)
to the end of do_write_request, after flush_dat/flush_idx complete, so a
successful write_all followed by a failed fsync no longer resets the
counter before the EIO is recorded. Repeated fsync EIOs now accumulate
toward the quarantine threshold as intended.
Recognize Windows storage-media failure codes ERROR_CRC (23) and
ERROR_IO_DEVICE (1117) in addition to Unix EIO (errno 5), so quarantined
heartbeat behavior is preserved on Windows. Mirrors the change in both
Go and Rust volume servers.
* fix(volume): preserve checkpoint EIO and clear streak on successful delete
maybe_checkpoint_index now returns whether the checkpoint succeeded;
the success-side EIO reset in do_write_request and do_delete_request
only fires when it did, so a checkpoint media failure is no longer
erased by the unconditional reset that followed it. do_delete_request
also gains the success reset that was lost when append_needle stopped
clearing the streak, so a successful delete still clears an earlier
failure streak.
is_storage_io_error now uses libc::EIO on Unix instead of a hard-coded
5, and the ECX binary-search read path gains a Windows fallback
(seek + read_exact) so the buffer is no longer zeroed on non-Unix
targets.
PR #11236 added a render-time guard that fails the chart when admin.ip
is non-loopback (default 0.0.0.0) and admin auth is not configured,
since weed admin 4.46 refuses to bind a non-loopback address without
authentication. Several pre-existing helm_ci.yml test cases enable
admin.enabled=true as part of "everything on" renders without a
password, so helm template now exits non-zero and the Verify template
rendering step fails.
Add admin.secret.adminPassword to the four render calls that turn on
admin without auth (IAM gRPC opt-in, NetworkPolicy EVERYTHING, egress
without kubeApiServer.cidrs, and the license ALL_ON dict), using the
same key ci/admin-values.yaml already uses.
* volume server: release the store lock before scrubbing EC volumes
`ec.scrub` makes a Rust volume server stop serving for the duration of the
scrub, and then kills its own gRPC connection:
error: rpc error: code = Unavailable desc = keepalive ping failed to
receive ACK within timeout
Measured on a 4.46 cluster (17 Rust volume servers on one host, ~520 volumes
and 53 EC volumes, --index=redb, EC 10+4). It reproduces against a SINGLE
node in 30-70s, in checksum, index and local modes, at -maxParallelization 1.
## Cause
The CHECKSUM arm of scrub_ec_volume reads every byte of every local shard
while holding the caller's store.read() guard:
let store = self.state.store.read().unwrap();
let ecv = store.find_ec_volume(vid)...?;
let (blocks, broken, errs) = ecv.checksum_scrub(); // GBs of I/O, lock held
VolumeServerState::store is a std::sync::RwLock, which is write-preferring.
The periodic heartbeat's collect_heartbeat_with_snapshot takes store.write()
and blocks; once that writer is pending, every later store.read() queues
behind it. Every HTTP handler takes store.read(), so the node serves nothing,
stops heart-beating, and cannot answer the scrub RPC's own keepalive - the
scrub kills the connection it is running on.
The INDEX and LOCAL arms have the same shape, and the node-wide scrub_volume
loop is worse: it held ONE guard across every volume on the node.
## Evidence
offcputime, off-CPU stacks >1s in a 30s window during a scrub:
futex_wait
seaweed_volume::server::heartbeat::collect_heartbeat_with_snapshot
- tokio-rt-worker
27967020 <- 27.97s blocked, of a 30s window
A single HTTP /status request issued 12s into a scrub, with 180s of patience,
was accepted and queued for 120 seconds, then served once the scrub released.
Thread states throughout: 1 D + 48 S. One thread working, 48 idle - not
executor starvation and no thread pileup, which is what a single lock holder
looks like.
Memory was tested and ruled out as the cause: the same scrub was run at
MemoryMax 3G, 8G and unlimited. With no limit there is no reclaim at all,
page cache grows freely to 22 GB, and the node still goes unresponsive at
t+30s. anon stays flat at 48-86 MB in every run.
## Fix
checksum_scrub, scrub_index and scrub_local gain plan types -
EcChecksumScrubPlan, EcIndexScrubPlan and EcLocalScrubPlan - snapshotted from
the volume under a brief guard. The handler builds a plan, drops the guard,
and runs the scan in spawn_blocking, off the async workers, since it is
synchronous CPU + file I/O either way.
A plan captures DESCRIPTORS, not paths. Resolving a path again after the
guard is dropped would let a writer that legitimately unlinks the files - the
heartbeat's delete_expired_ec_volumes, which reaches EcVolume::destroy(), or
volume_ec_shards_delete - surface an intentional removal as "scrub read
error: No such file or directory" and put the volume in broken_volume_ids. A
descriptor outlives the name.
For the shards it duplicates the handle the mounted EcVolumeShard already
holds (try_clone_file), which is what Go does: ChecksumScrub reads through
shard.ReadAt (weed/storage/erasure_coding/ec_volume_scrub.go:71), never
through a path. That also inherits open_volume_file's O_NOATIME and drops a
dead branch - the old code built {base}.ec{id}.v{gen} for a non-zero
generation, a name nothing in this tree writes. dup shares the kernel offset,
so shard reads stay positional; the .ecx gets a fresh open instead, since
check_index_file seeks.
FULL/READS is unchanged here: it already released the guard across the index
walk, and still re-takes it per needle in store_ec::scrub_snapshot_under_lock
for that needle's local shard intervals - short holds, many of them.
scrub_volume now takes the read guard PER VOLUME instead of across the whole
loop, so the heartbeat can land between volumes. Its per-volume work still
runs under the guard; Volume needs an equivalent plan to fix that properly,
left as a follow-up and noted in the code.
## A failed scrub task must not take the whole RPC down
Moving the scans into spawn_blocking changed where a panic lands. It no
longer unwinds inside the handler's own future; it comes back as a JoinError
at the .await, and all four join points sat behind a `?`. So one bad volume
out of six hundred returned Err from the entire handler: the
broken_volume_ids, broken_shard_infos and details already gathered for the
other 599 were dropped, and emit_scrub_metrics - the only writer of
SCRUB_LAST_TIME_SECONDS, SCRUB_VOLUME_FAILURES and SCRUB_SHARD_FAILURES - was
never reached, so the staleness alert kept firing while real corruption went
unreported.
And there is a reachable panic behind it. EcLocalScrubPlan::run() sized its
reassembly buffer with
Vec::with_capacity(get_actual_size(size, version) as usize)
which for any negative size that is not the -1 tombstone skipped above is a
capacity-overflow abort. Mode 3 (LOCAL) is the default of `weed shell
ec.scrub`, and a scrub is what you point at an index you already suspect, so
an arbitrary i32 in a .ecx size field is in-scope input. The buffer is
Rust-only - Go appends to a nil slice and has no capacity hint here. Guard on
`want <= 0` and fall through with an empty buffer: locate_data returns no
intervals for a non-positive size, read stays 0, and the existing
`read != want` error reports the row exactly as Go does.
Each join point now records the failure against its own volume and continues.
A panic is evidence about the volume and counts as broken; a non-panic
JoinError is not - spawn_blocking only reports one when the runtime is going
down, the volume was never scanned, and counting it would put a false
corruption into SCRUB_VOLUME_FAILURES. total_volumes moves before the join in
modes 1, 3 and 4 (2|5 already counted there) so a failed join cannot silently
shrink it. Mode 2|5's verify_ec_shards join is the one that must not
`continue`: the needle walk above has already produced findings for that
volume.
The tombstone guard stays is_tombstone() on purpose. ScrubLocal in
ec_volume_scrub.go:228 skips only IsTombstone(), while the distributed walk
in store_ec.go:516 skips all IsDeleted() - the asymmetry is Go's, and both
Rust walks mirror their own counterpart.
## Both servers: a node-wide scrub skips a volume that vanished mid-run
Releasing the lock makes the volume set legitimately mutable during a scrub,
so a node-wide run can reach a volume that has since been unmounted. That is
not a scrub failure. A node-wide run now logs and skips it; an explicitly
requested volume id still returns NotFound. The Go server is changed the same
way, so both implementations answer the same shell command identically.
mark_broken_volumes_readonly tolerates the same teardown one step later,
instead of throwing away the whole scrub report.
## Test
test_scrub_plans_are_self_contained_and_match_direct_call drops the EcVolume
and runs both plans on another thread, asserting the results match the direct
calls. A plan that borrowed from EcVolume could do neither, so the test stops
compiling if the snapshot regresses to a borrow.
test_scrub_plans_survive_files_removed_after_snapshot unlinks every shard and
the .ecx after the plans are built, then asserts the results still equal the
direct call. Against a path-resolving version it fails with all 14 shards
reported as "No such file or directory".
test_local_scrub_plan_reports_negative_size_ecx_row rewrites a .ecx row's
size to -1000 and runs the local plan on another thread, so the join is the
assertion - that thread is the spawn_blocking whose panic used to fail the
RPC. Without the capacity guard it fails with "capacity overflow"; with it,
the row is reported.
The Go tests cover both halves of the vanished-volume rule for volumes and EC
volumes.
517 lib tests pass, plus 34 across the other targets (`cargo test`).
`go test ./weed/server -run Scrub` passes.
## Known remaining, not fixed here
`ec.scrub -volumeId=N` is still fanned out to every node, and a node that
holds no shard of N returns NotFound, so the shell command errors even when
the nodes that do hold shards scrub cleanly. That is a shell-side fan-out
question rather than a volume-server one, and both servers keep the existing
behaviour for an explicitly requested id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DvHoW85w6SNKNBPvrqLMmK
* scrub: discard checksum block count from total_files; capture .ecx fd for FULL walk
Two review fixes:
1. CHECKSUM arm: plan.run() returns blocks scanned, not a file count.
Go discards it (_, shardInfos, serrs = v.ChecksumScrub()) so TotalFiles
stays a needle/file count. The Rust arm was adding it to total_files,
inflating the count. Discard it to match Go.
2. FULL/READS (scrub_ec_volume_distributed): the needle walk reopened the
.ecx by PATH after the store guard was released, so a concurrent teardown
that unlinks or replaces the .ecx (heartbeat delete_expired_ec_volumes,
volume_ec_shards_delete) could surface an intentional removal as a scrub
error or mix index generations within one scrub. Capture a second .ecx
descriptor under the guard (the index plan handle is consumed by its own
structural walk, and both seek) and read through it instead -- the same
descriptor-outlives-name invariant the checksum plan shard handles use.
* scrub: bind FULL/READS walk to one encode generation
Address Devin review: after capturing the .ecx descriptor under the guard,
scrub_snapshot_under_lock still re-resolves the volume by id per needle, so
a teardown-and-remount of the same vid between two rows would apply the
captured .ecx offsets to a replacement volume's shards -- falsely reporting
corruption.
Capture the volume's encode_ts_ns (encode-run identity) in Phase A and pass
it to scrub_snapshot_under_lock. If the mounted volume's encode_ts_ns no
longer matches, abort the walk like a mid-scan unmount instead of mixing
generations within one scrub.
* scrub: run FULL/READS index scan in the blocking pool
Address CodeRabbit review (5147767192): index_plan.run() reads the whole
.ecx synchronously, so running it on the async executor worker could block
unrelated RPC work handled on the same executor. Move it into spawn_blocking,
matching the treatment the CHECKSUM/LOCAL arms already give their plans. A
join failure (panic/cancellation) is reported as a seed error so the
per-volume findings below are not silently dropped.
* scrub: move ecx walk to blocking pool, classify join errors, guard encode_ts_ns==0
Three CodeRabbit review fixes (5148034447):
1. Move the FULL/READS needle walk (walk_index_file over the captured ecx
descriptor) into spawn_blocking. It reads the full .ecx synchronously and
was still running on the async executor worker, the same blocker the
index_plan.run() fix in the previous commit addressed.
2. Preserve JoinError classification in both spawn_blocking join points in
scrub_ec_volume_distributed. A panic is evidence about the volume and
counts as broken; a cancellation only happens at runtime shutdown, the
volume was never scanned, and returning it as an error would put a false
corruption into broken_volume_ids (the FULL/READS arm marks the volume
broken on any non-empty errs). Panics return an error; cancellations
return clean.
3. Do not treat encode_ts_ns == 0 as a verified generation match. The .vif
assigns 0 when it carries no encode-run identity (legacy/pre-feature
volumes), so 0 == 0 would accept a teardown-and-remount and apply the old
.ecx offsets to the replacement volume's shards. Only enforce the
generation check when the captured identity is non-zero; when it is zero,
fall back to the pre-check behavior (no generation binding) rather than
aborting a scrub that was already running without the guard.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The metadata-subscription readahead channels were sized for a
low-throughput era and now bottleneck replay catch-up:
- ReadPersistedLogBuffer's readaheadSize was 1024 entries: the
background visitor fills the channel, then blocks on the consumer's
gRPC Send, so volume-server I/O for the next log file never overlaps
with delivery of the current one. Each disk pass takes longer, and
the subscribe loop re-lists log files (ListDirectoryEntries on the
filer store) more often to drain the same backlog. Raised to 8192
so the reader stays ahead of the consumer through a full log file's
worth of entries.
- readFilersMerged's logEntryChannelSize was 512 entries per filer
stream: the same serialization on the client side, where weed mount
(chunk mode) reads persisted log chunks directly from volume
servers. A small channel means the producer stalls on the merge
consumer's processEventFn, and the next log file's chunks are never
fetched ahead. Raised to 4096 so volume I/O overlaps with event
delivery.
The wider buffers keep the producer goroutines reading through a full
log file while the consumer is still processing the previous one,
turning serial read→process→read into pipelined read∥process. This
cuts the per-pass wall time that drives filer store listings and
volume-server round-trips, reducing filer workload under backlog
catch-up (e.g. CSI deployments where ~200 mounts reconnect on filer
restart).
* admin: allow insecurely binding to any IP if -allowInsecureNoAuth is set
* admin: rename -allowInsecureNoAuth to -allowInsecureBind
The new flag name is shorter and clearer: it describes what is being
allowed (an insecure bind to a non-loopback address) without the
redundant "NoAuth" suffix.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: pass -ip to admin so StatefulSet becomes Ready on 4.46
Since weed admin 4.46 changed its default listen address from all
interfaces to loopback (127.0.0.1), the admin StatefulSet template never
passed -ip, so the admin server bound to loopback only. The chart's httpGet
readiness/liveness probes dial the pod IP, not loopback, so the probes
never succeeded and the admin StatefulSet stayed 0/1 forever — breaking
upgrades with helm --wait or GitOps controllers.
Add an admin.ip value (default "0.0.0.0", restoring the pre-4.46
behaviour) and render it as -ip. A non-loopback bind requires
authentication, so fail at render time when admin.ip is non-loopback and
neither admin.secret.adminPassword nor admin.secret.existingSecret is
set, instead of letting the pod crash-loop. Document the value and add a
chart-testing CI values file.
Bumps chart to 4.46.1.
Fixes#11234
* helm: address review feedback on admin bind validation
Align the chart's loopback classification with weed admin's isLoopbackIp
(net.ParseIP + IsLoopback): the whole 127.0.0.0/8 range and ::1 are
loopback; localhost and wildcard addresses are non-loopback, matching
the binary. Previously the exact-string check rejected valid loopback
addresses like 127.0.0.2 while permitting localhost (which the binary
treats as non-loopback).
Recognize WEED_ADMIN_PASSWORD supplied via admin.extraEnvironmentVars /
admin.secretExtraEnvironmentVars as authentication, since weed admin
picks it up through viper's AutomaticEnv. Previously such deployments
were wrongly rejected at render time.
Remove [https.admin] mTLS from the validation message and docs: the
chart only generates [grpc.admin] (gRPC mTLS), not [https.admin] (HTTP
mTLS), so mentioning it as an alternative was misleading.
Document that the -ip flag requires SeaweedFS 4.46 or newer, so pinning
admin.imageOverride to an older image is not supported with this chart.
Extracted the loopback and auth checks into reusable helpers
(seaweedfs.admin.isLoopbackIp, seaweedfs.admin.authEnabled) following
the existing seaweedfs.filer.mysqlEnabled pattern.
* helm: tighten loopback classification to reject malformed 127.x addresses
Use regexMatch instead of hasPrefix for the IPv4 loopback check so
malformed values like "127.not-an-ip" are not accepted as loopback
(net.ParseIP returns nil for them, so weed admin treats them as
non-loopback). Also recognize the expanded IPv6 loopback form
"0:0:0:0:0:0:0:1" in addition to "::1", matching net.ParseIP behavior
for the two common representations.
* filer/postgres: create default filemeta table on startup
The postgres filer store hardcoded CreateTableSqlTemplate to empty and
never created the filemeta table, unlike postgres2/mysql2/sqlite which
all create it during Initialize. Users had to create the table manually
or the filer would crash loop with "relation filemeta does not exist".
Read the createTable config option (same as postgres2), default to
DefaultCreateTableQuery when unset, and execute CREATE TABLE IF NOT EXISTS
on the default table after the connection pool is established.
SupportBucketTable stays false so per-bucket table creation remains a
no-op; only the shared filemeta table is created, via a direct ExecContext
since AbstractSqlStore.CreateTable short-circuits without bucket support.
* filer/postgres: accept boolean createTable = true/false
viper reads a TOML boolean as the string "true"/"false" via GetString, so
createTable = true was being used as a SQL template and failed. Add
ResolveCreateTableQuery to normalize the value: true and empty select the
default template, false disables table creation, anything else is a custom
template. Both postgres and postgres2 now use it, and both skip the CREATE
TABLE call when the resolved template is empty.
* scaffold: document createTable option for postgres filer store
Replace the commented-out CREATE TABLE SQL in the [postgres] scaffold with
a createTable config hint, matching the [postgres2] section. Users no longer
need to manually create the filemeta table before starting the filer.
* filer/postgres: make createTable opt-in for postgres, keep postgres2 default
The previous commit defaulted postgres to create the filemeta table even
when createTable was unset, which could break existing deployments whose
DB user lacks CREATE TABLE privileges. ResolveCreateTableQuery now returns
empty for an unset value so postgres only creates the table when
createTable is explicitly true or a custom template — preserving the
prior no-DDL behaviour for existing configurations.
postgres2 keeps its existing always-create default: it defaults an empty
resolved value to DefaultCreateTableQuery, and only skips when
createTable is explicitly false.
* filer/postgres2: simplify createTable handling, document all modes
Drop the false opt-out from postgres2 — it only skipped the default table
while per-bucket CreateTable still ran, leaving restricted DB roles broken
on bucket access. postgres2 now accepts true the same way (defaulting to
DefaultCreateTableQuery) and keeps its existing always-create behaviour
for every other value, matching the original semantics.
The scaffold comment now documents true/false/custom for the postgres
section so users know false (or unset) is the backward-compatible default.
* filer/postgres2: normalize false via ResolveCreateTableQuery
postgres2 only handled "" and "true", leaving createTable = false as the
literal string "false" which CreateTable then executed as invalid SQL.
Route it through ResolveCreateTableQuery (which maps false to empty) and
default the empty result to DefaultCreateTableQuery, so false is treated
the same as unset for the bucket-aware store.
* filer/postgres2: honor createTable = false for default table
postgres2 treated false the same as unset and always created the default
filemeta table, failing startup for restricted DB roles that explicitly
opted out. Track the original false value before ResolveCreateTableQuery
collapses it to empty, and skip the default CreateTable call when set.
Per-bucket table creation is unaffected — it is a runtime requirement of
the bucket-aware store. Users who need to suppress all DDL should use the
postgres (non-bucket) store with createTable unset.
* filer/postgres2: disable bucket tables when createTable = false
Setting SupportBucketTable = false when createTable is explicitly false
makes AbstractSqlStore.CreateTable a no-op (it already returns nil when
SupportBucketTable is false), so neither the default filemeta table nor
per-bucket tables are created. The template stays empty and no DDL runs,
honouring the opt-out for restricted DB roles. All data routes to the
pre-provisioned filemeta table, matching the postgres (non-bucket) store.
* filer: suppress DDL without disabling bucket routing
Setting SupportBucketTable = false when createTable = false also disabled
per-bucket routing, hiding objects in pre-provisioned per-bucket tables.
Keep SupportBucketTable true and instead skip the CREATE TABLE execution
when the resolved template is empty.
GetSqlCreateTable now returns empty for both postgres and mysql SQL
generators when CreateTableSqlTemplate is empty, and
AbstractSqlStore.CreateTable skips the ExecContext call when the SQL is
empty. This preserves bucket routing while suppressing all DDL for users
who explicitly set createTable = false and pre-provision their tables.
* filer: add SkipDDL to suppress CREATE and DROP without disabling routing
createTable = false with SupportBucketTable = true preserved bucket
routing but deleteTable still executed DROP TABLE on bucket deletion,
dropping externally managed tables. CanDropWholeBucket also returned
true, so the S3 layer tried whole-table drops instead of row-by-row
deletes.
Add a SkipDDL flag to AbstractSqlStore, independent of SupportBucketTable.
CreateTable and deleteTable both skip when SkipDDL is set, and
CanDropWholeBucket returns false so bucket deletion falls back to
row-by-row metadata deletes. postgres2 sets SkipDDL when createTable is
explicitly false — bucket routing is preserved, no DDL runs.
* filer: fall back to row-by-row delete when CanDropWholeBucket is false
DeleteFolderChildren took the whole-table drop path whenever the path was
a bucket root, even when SkipDDL made deleteTable a no-op. The no-op
returned nil, the caller returned early, and rows inserted after the
recursive enumeration survived the bucket deletion.
Gate the whole-table drop on CanDropWholeBucket so the row-by-row
DeleteFolderChildren SQL runs when SkipDDL is set, removing all metadata
without issuing DROP TABLE.
* admin: add IsPolicyAttached helper to detect user/group attachments
Introduces AdminServer.IsPolicyAttached, which lists the users and groups
that still have a managed policy attached, reusing the existing credential
manager ListUsers / ListAttachedUserPolicies / ListGroups / GetGroup
methods. This is the building block for rejecting policy deletion while a
policy is still referenced, so deleted policy names stop lingering in a
user attached policy names list (issue #11225).
* admin: reject IAM policy deletion while still attached
Guards AdminServer.DeletePolicy with the new IsPolicyAttached check and
returns the typed ErrPolicyStillAttached error when the policy is still
referenced by a user or group. This matches AWS IAM and the existing IAM
API handler behavior, fixing the stale reference where a deleted policy
name kept showing up in a user attached policy names list (issue #11225).
* admin: return 409 Conflict when deleting an attached IAM policy
The admin UI DeletePolicy handler now maps ErrPolicyStillAttached to HTTP
409 Conflict instead of 500, so the dashboard can surface the attachment
conflict to the user rather than reporting a generic server error.
* admin: skip vanished groups when checking policy attachments
IsPolicyAttached now treats a group that disappears between ListGroups
and GetGroup (credential.ErrGroupNotFound) as no longer attached instead
of failing the whole deletion with HTTP 500, matching the IAM API handler
which skips vanished groups.
* test: assert policy state after deletion paths
Strengthen GetPolicy assertions in the policy deletion tests to check the
returned policy is non-nil after a rejected deletion and nil after a
successful one, not just that no lookup error occurred (GetPolicy returns
nil, nil when a policy is absent).
* fix(s3api): use direct peer IP for aws:SourceIp in bucket policy engine
extractSourceIP in the bucket-policy engine trusted X-Forwarded-For and
X-Real-Ip whenever the TCP peer looked private (loopback/RFC1918/link-local),
with no configurable trusted-proxy allowlist. In containerized deployments
the gateway peer is almost always private, so any caller reaching it directly
or from a co-located workload could spoof aws:SourceIp and bypass
IpAddress/NotIpAddress bucket-policy restrictions.
Always return the direct peer address (r.RemoteAddr), matching AWS S3
semantics. Remove the now-unused isPrivateIP helper and header-trust branch.
Update TestExtractConditionValuesFromRequestSourceIPPrecedence to assert the
peer IP is used regardless of forwarding headers, and add regression tests
TestExtractSourceIP_IgnoresForwardedHeaders and
TestExtractSourceIP_EnforcesIPRestrictionPolicy.
* fix(s3api): use direct peer IP for aws:SourceIp in IAM role/session policies
The IAM middleware's extractSourceIP trusted X-Forwarded-For and X-Real-IP
whenever the TCP peer looked private (loopback/RFC1918/link-local), with no
configurable trusted-proxy allowlist. In containerized deployments the gateway
peer is almost always private, so any caller reaching it directly or from a
co-located workload could spoof aws:SourceIp and bypass IpAddress/NotIpAddress
conditions on role and session policies (IsPrincipalActionExplicitlyDenied).
Always return the direct peer address (r.RemoteAddr), matching AWS S3
semantics. Remove the now-unused isPrivateIP helper, privateNetworks table,
and its init().
Update TestRequestContextExtraction and TestIPBasedPolicyEnforcement to assert
the peer IP is enforced regardless of forwarding headers, and add regression
test TestUserInlinePolicySourceIpCondition_IgnoresForwardedHeaders.
* iam: expose tri-state result from attached policy evaluation
evaluateIAMPolicies returned a bool that collapsed explicit Deny and
no-match into a single false, so the authorization path could not tell
"policies forbid this" from "policies say nothing". Introduce
evaluateAttachedIAMPolicies returning Allow/Deny/NoMatch and keep
evaluateIAMPolicies as a bool projection for existing callers. This is
preparation for unioning native permissions with attached policies while
preserving deny-always-wins.
* iam: preserve native Admin when IAM policies are attached
Attaching an IAM policy routed authorization exclusively to the attached
policies, dropping the identity native permissions. A user with native
Admin lost all access after attaching a non-granting policy, and stayed
locked out if that policy was deleted without being detached first
(#11226).
Treat a native bare Admin grant as a permission floor that survives
attached policies: when the attached policies do not explicitly allow,
fall back to isAdmin() on the attached-policy path, and on the IAM
integration path allow unless an attached policy explicitly denies.
Explicit Deny still wins on both paths.
Only bare Admin is consulted because inline policies flatten lossily into
Actions (dropping conditions), so scoped actions are not unambiguously
native and must keep flowing through the policy engine.
* iam: regression tests for native Admin surviving attached policies
Reproduces issue #11226:
- TestNativeAdminSurvivesAttachedPolicy: a user with native Admin keeps
Write access after attaching a policy that does not grant it.
- TestNativeAdminSurvivesDeletedPolicy: the same user keeps Write access
after the attached policy is deleted without being detached.
- TestAttachedPolicyExplicitDenyOverridesNativeAdmin: an explicit Deny in
an attached policy still constrains a native admin (deny-always-wins).
* iam: apply native Admin floor before IAM principal validation
The native Admin floor in authorizeWithIAM ran after the auth-path
switch, which denies when no session principal or PrincipalArn is
present. An Admin identity without a PrincipalArn (no session token)
was therefore denied before the floor executed. Move the floor ahead of
the switch and derive the principal for its explicit-deny check with
buildPrincipalARN, which already handles identities without a
PrincipalArn. Adds a regression case for an Admin identity with an
empty PrincipalArn.
Addresses CodeRabbit review feedback on PR #11232.
* fix(filer): bound retained reader cache buffers by bytes
* test(filer): keep in-flight downloads during cache trimming
* feat(mem): expose pooled allocation capacity for byte reservations
* fix(mount): share a configurable reader buffer budget across files
* fix(filer): release failed prefetch slots and memory reservations
* feat(mount): expose a soft Go runtime memory limit
* docs(filer): restore shared-download rationale in startCaching
The one-line comment replacing the original context.Background() explanation was too thin for readChunkAt to cross-reference shared resource semantics. Restore a concise note on why request cancellation must not abort a download shared by concurrent readers.
* test(filer): loosen reader cache test deadlines to 5s
Three tests used 1-second deadlines that can flake on CI under load:
TestReaderCacheBudgetInFlight, TestReaderCacheEvictionDoesNotHoldCacheLock,
and TestReaderCacheFailedPrefetchReleasesBudget. Increase to 5 seconds.
* test(filer): cover re-read after reader cache eviction
Add TestReaderCacheReReadAfterEviction: reads chunk 'a', reads chunk 'b'
(evicting 'a' via budget pressure), then re-reads 'a' and asserts a
fresh download returns correct data. Verifies the core correctness
property that eviction never exposes missing or stale data to readers.
Allow normal post-tag proxy propagation before dispatching downstream releases, while preserving the check that prevents them from pinning the previous commit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* iam: bind CreateServiceAccount target to caller in AuthorizeIamAction
A non-admin holding iam:CreateServiceAccount could pass an arbitrary
ParentUser and mint a service account for any identity, inheriting that
identity permissions. Add a self-target category so a granted non-admin
may only target their own identity; admins remain unrestricted.
* iam: authorize CreateServiceAccount against its ParentUser target
AuthIamManagement passed UserName as the authorization target for every
action, so CreateServiceAccount was authorized with an empty target and
the self-target binding never saw the caller-supplied ParentUser. Pass
ParentUser for that action so the binding takes effect on the live path.
* iam: test CreateServiceAccount binds target to caller
Regression test: a non-admin holding iam:CreateServiceAccount may target
itself but is denied targeting another identity; admins remain
unrestricted.
* iam: authorize CreateServiceAccount against ParentUser on the S3 port
UnifiedPostHandler passed UserName as the authorization target for every
IAM action, so CreateServiceAccount was authorized with an empty target
on the S3-port route and the self-target binding never saw the caller
ParentUser. Extract iamTargetUserName (ParentUser for CreateServiceAccount,
UserName otherwise) and use it from both IAM dispatch surfaces so the
binding applies on the live S3-port path as well as the standalone iam
server.
* iam: test CreateServiceAccount ParentUser binding on the S3 port
End-to-end regression test through UnifiedPostHandler: a non-admin
holding iam:CreateServiceAccount is denied (403) when targeting another
identity and passes authorization when targeting itself.
* fix issue-11214
* fix(filer): cancel sibling manifest reads on failure
* fix(filer): scope manifest cancellation to read batch and propagate context to encrypted reads
Address PR review comments on #11215:
- Scope cancellation to each parallel read batch instead of the resolver-wide
context, so a later manifest failure does not cancel recursive work for an
earlier successful manifest (CodeRabbit #3952446554).
- Propagate the resolver context through GetAuthenticatedWithContext so
encrypted sibling reads observe cancellation and stop promptly when another
manifest fails (Greptile #3952422343).
- Use net.ListenConfig.Listen with an explicit context in the test fixture to
satisfy the noctx linter (CodeRabbit #3952111696).
- Add regression tests for encrypted sibling cancellation and for preserving
earlier manifest children on later failure.
* fix(filer): return known manifest errors without blocking on earlier children
Address Greptile review comment on #11215:
After all parallel reads complete, pre-scan slots for the first real
(non-internal-cancel) error before recursing into earlier manifests'
children. If a later manifest already failed, return its error promptly
with data chunks already in hand, instead of blocking on recursive
network reads of earlier manifests' children.
Updated the regression test to verify the error returns within 1 second
when an earlier manifest's child has a 2-second delay, and that the
child is never loaded.
* fix(filer): filter partial child chunks by requested range on error path
Address CodeRabbit review comment on #11215:
The pre-scan error path appended non-manifest child chunks from earlier
manifests without applying the [startOffset, stopOffset) overlap check
used for top-level chunks. A child outside the requested range could be
returned in dataChunks alongside the later manifest's error.
Apply the same range predicate before appending. Add regression test
with an out-of-range child chunk.
* fix(filer): buffer job channel and abort submission on batch cancellation
Address Greptile review comment on #11215:
- Use a buffered job channel (capacity 128) so submission does not block
when all workers are busy. This ensures a promptly-failing manifest is
always queued and can cancel stalled sibling reads once a worker picks
it up, instead of blocking the caller on the unbuffered channel send.
- Add batchCtx.Done() to the submit select so submission aborts promptly
when the batch is already cancelled by a sibling failure.
- Add regression test with 5 manifests (4 stalled + 1 failing) verifying
the failing job is queued and picked up after a stalled worker is freed.
* fix(filer): avoid double WaitGroup decrement on batch cancellation in submit
Address Devin review comment on #11215:
When batchCtx.Done() fired in submit, it called job.done.Done() and
returned false. The caller in resolve also called reads.Done() on the
same WaitGroup, causing a double decrement that would panic with a
negative counter.
Fix: submit sets the result error but does not decrement the WaitGroup.
The caller always owns the decrement and skips overwriting the result
when submit already set it.
* fix(filer): overflow execution when job queue buffer is full
Address Greptile follow-up review comment on #11215:
With a 128-entry buffer, if more than 132 in-range manifests (4 workers +
128 buffer) stall at one level, a promptly-failing manifest beyond the
buffer cannot be submitted and cannot cancel the stalled reads.
Fix: when the buffer is full, run the job directly in a goroutine instead
of blocking on the channel send. This only triggers for >132 manifests at
one level (exceedingly rare), so the bounded concurrency guarantee (4
workers) holds for all normal workloads. Extracted executeJob method
shared by both workers and overflow goroutines.
* fix(filer): bound overflow execution with a semaphore
Address Devin review comment on #11215:
The unbounded overflow goroutines could create thousands of concurrent
reads for large files, defeating the four-worker resource bound.
Fix: add a semaphore (capacity = maxChunkManifestResolveWorkers) that
overflow goroutines must acquire before doing the read. While waiting for
the semaphore, they also watch batchCtx and r.ctx so they exit promptly
on cancellation. Total concurrency is now bounded to 2 * workers (4
workers + 4 overflow) in the degenerate case.
* refactor(http): add ctx to GetAuthenticated signature instead of new function
Reuse the existing GetAuthenticated name by adding ctx as the first
parameter, matching the pattern of ReadUrl, ReadUrlAsStream, and
RetriedFetchChunkData. Removes the GetAuthenticatedWithContext wrapper.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* admin: support setting volume read-only and read/write modes
* admin: address PR review on volume access-mode persistence
Reject trailing JSON values in the SetVolumeReadOnly handler so
requests like {"read_only":true}{} no longer pass validation, and add
a trailing-value case to the invalid-request test.
Propagate .vif persistence failures through the access-mode chain.
PersistReadOnly now returns the SaveVolumeInfo error and rolls back
the in-memory volumeInfo on failure; Store.MarkVolumeReadonly and
Store.MarkVolumeWritable propagate that error and roll back their
noWrite flags, so the API reports failure instead of success while
restart would revert the mode.
* admin: make .vif persistence atomic and preserve error chain
SaveVolumeInfo now writes to a .vif.tmp file, syncs it, renames it
over the target, and fsyncs the directory. A write/sync/close failure
leaves the existing .vif intact, so the PersistReadOnly in-memory
rollback matches the durable state instead of diverging from a
partially written file that restart would apply.
Switch the error wrappers in PersistReadOnly, MarkVolumeReadonly, and
MarkVolumeWritable from %v to %w so callers can use errors.Is and
errors.As to classify persistence failures.
* admin: treat post-rename dir fsync failure as a warning
After os.Rename commits the new .vif, the on-disk file already holds
the requested mode. A directory fsync failure only risks losing the
rename across a crash; returning an error here would make
PersistReadOnly roll back in-memory state while the durable file keeps
the new mode, splitting the replica. Log the failure as a warning
instead, matching the best-effort nature of FsyncDir (already skipped
on Windows).
* admin: distinguish post-rename durability failures and use unique temp files
SaveVolumeInfo now uses os.CreateTemp for the staging file, preventing
concurrent saves for the same volume from colliding on a shared .tmp
path.
A directory fsync failure after os.Rename returns a
NotCrashDurableError instead of being silently swallowed. The rename
already committed the new metadata to disk, so PersistReadOnly,
MarkVolumeReadonly, and MarkVolumeWritable skip the in-memory rollback
for this error type (keeping state aligned with the durable file) while
still propagating the failure to the API. Pre-commit failures continue
to roll back as before.
* admin: continue post-commit work after NotCrashDurableError
MarkVolumeWritable now clears the EIO quarantine and the gRPC handlers
(makeVolumeReadonly step 3, makeVolumeWritable master notification)
proceed with their post-commit work when SaveVolumeInfo returns a
NotCrashDurableError, instead of aborting and leaving the volume
unavailable or the master unaware of the mode change. The durability
warning is still propagated to the API caller. Pre-commit failures
continue to abort early as before.
* admin: handle NotCrashDurableError in tier and EC callers
VolumeTierMoveDatFromRemote and VolumeEcShardsGenerate now check for
NotCrashDurableError from SaveVolumeInfo. When the rename has already
committed the new .vif, they continue with their post-commit work
(backend switch, remote deletion, keeping generated EC shards) instead
of aborting and leaving the on-disk metadata inconsistent with the
file layout. The durability warning is logged for the operator.
* build(deps): bump github.com/twmb/avro from 1.7.2 to 1.8.0
Bumps [github.com/twmb/avro](https://github.com/twmb/avro) from 1.7.2 to 1.8.0.
- [Commits](https://github.com/twmb/avro/compare/v1.7.2...v1.8.0)
---
updated-dependencies:
- dependency-name: github.com/twmb/avro
dependency-version: 1.8.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* iceberg: adapt to twmb/avro v1.8.0 and iceberg-go defensive copies
avro v1.8.0 changes Schema.Root() to return *SchemaNode, which breaks
iceberg-go v0.6.0's internal avro_schemas.go. The fix (apache/iceberg-go#1843)
is only on iceberg-go's main branch, unreleased, so bump iceberg-go to
that commit (c210509) alongside the avro bump.
That iceberg-go revision also changes two behaviors seaweedfs worked
around:
- It now infers a manifest list's format version from the embedded
writer schema, so a list missing the "format-version" header entry
(DuckDB's shape is read as v2, not v1. ReadManifestList's header
patching is now a redundant safety net; tests updated to expect v2.
- It returns defensive copies from DataFile.Partition(), so the
ReadManifest shim's in-place partition normalization was silently
discarded. Rebuild the entry through NewDataFileBuilder when any
partition value is normalized, copying every other DataFile field so
manifest round-trips are preserved.
- It converts day-transform partitions to iceberg.Date on read
(applyDayTransformDates), so the day-partition cases the shim and
tests guarded now convert without help; tests updated to expect
iceberg.Date from the raw read.
EOF
)
* iceberg: accept assert-ref-snapshot-id without snapshot-id
iceberg-go's new nullableInt64 parser rejects an assert-ref-snapshot-id
requirement whose "snapshot-id" field is absent from the JSON, even
though the Iceberg REST spec makes it optional (null means the ref must
not already exist). v0.6.0 used a plain *int64, so absent was nil and
accepted. ClickHouse sends the requirement without snapshot-id when
asserting a branch does not yet exist, so its writes fail with
"missing required field \"snapshot-id\"".
normalizeRequirements splices an explicit null into any
assert-ref-snapshot-id requirement missing the field before handing
the JSON to iceberg-go's parser, restoring the v0.6.0 behavior across
both iceberg-go versions.
* iceberg: fix v1 block_size_in_bytes default in rebuilt manifest entries
rebuildManifestEntry set block_size_in_bytes to 0, but the v1 manifest
schema requires the default of 64 MiB ("Always write default in v1").
The original value is not exposed on the DataFile interface, so use the
spec default. Also clarify the fallback comment to note that empty
(zero-record / zero-byte) files also trigger it, not just a nil spec.
Add a round-trip test that writes a rebuilt entry as v1 and verifies
block_size_in_bytes is 64 MiB via Avro decoding.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust volume: insert redb rebuilds in needle-id order
Unlink the .rdb before create (create does not truncate). Collapse
last-write-wins, then insert live keys sorted so 4.2.0 packs leaves.
* rust volume: rebuild redb from a BTreeMap and clear leftover keys
Peak rebuild memory is one ordered map instead of HashMap + Vec +
stable-sort scratch. Unlink stays best-effort: if it fails, retain
clears the leftover table before sorted insert. Compute idx metrics
before the write so a read error does not unlink a committed .rdb.
* rust volume: drop the extra redb read transaction on put/delete
put uses insert()'s previous value. delete gets then inserts the
tombstone in the same write transaction. Truncate the .idx row on
any failed redb write after the append.
* rust volume: unpack redb blobs through packed_to_needle_value
save_to_idx, ascending_visit, and collect_entries used the same
length-check copy as get. Route them through the helper so a
wrong-length value is absent everywhere, not a panic.
* rust volume: quick-repair redb on durable checkpoints
set_quick_repair(true) on the durable checkpoint transaction so an
OOM-killed volume server opens without a full-file repair scan.
* rust volume: reopen redb from .idx on non-poisoned commit error
redb 4.2.0 can make a Durability::None commit visible before returning
Err(CommitError::Storage(..)). In that state the database refuses further
write transactions, so truncating the .idx row (the old behavior) would
leave a redb-only put or tombstone that the stored idx_size makes the
reload skip.
Distinguish CommitError::TransactionPoisoned (txn rolled back, db still
usable -- truncate the orphan .idx row as before) from other commit errors
(change may be visible, db refuses writes -- keep the .idx row, close the
database, and reopen from .idx to repair redb's internal state).
db becomes Option<Database> so reopen_from_idx can drop the old file lock
before load_from_idx opens the same path. rdb_path, version, and
cache_bytes are stored so the reopen uses the same configuration.
* rust volume: truncate .idx row when redb is closed in put
put appends the .idx entry before acquiring the write transaction.
When db_or_err() fails (db is None after a failed reopen), the ?
returned without calling truncate_idx_to_offset, so a write reported
failed remained in the authoritative .idx and was replayed on restart.
Handle db_or_err() explicitly and truncate the orphan .idx row before
returning the error, matching the existing handling for begin_write,
open_table, and insert failures.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
SetVolumeCrowded mutated the crowded map under accessLock.RLock(), while
GetWritableVolumeCount reads the same map under RLock() on the Assign hot
path. Two concurrent RLock holders with one writing and one reading the
map triggers a fatal "concurrent map read and map write" that kills the
master process (unrecoverable, bypasses recover).
Take the write lock in SetVolumeCrowded instead. This event path is a
low-frequency single consumer driven by the crowded-volume event loop,
and every other mutation of crowded already holds Lock(); setVolumeCrowded
takes no nested locks, so there is no deadlock path. The hot readers
(GetWritableVolumeCount, CloneWritableVolumes) keep using RLock.
Adds a -race regression test that fails (race detected) on the old RLock
and passes with the write lock.
Fixes#11211
* fix(log_buffer): stop closing a notification channel another reader still holds - #10810
The report blames the polling loop for the busy spin, but that loop is
not what burns the core. LogBuffer keeps one notification channel per
subscriberID, and UnregisterSubscriber closes it. Two registrations that
share a subscriberID share that channel, which happens whenever a client
opens a second stream or an old stream has not yet noticed it was
replaced, so the first unregister closes a channel the other reader is
parked on. A closed channel makes every receive in
awaitNotificationOrTimeoutFor return instantly, and that reader then
spins at full speed for the rest of its life.
Subscriptions are now reference counted. Registering an existing
subscriberID hands back the same channel and raises the count; the
channel is only closed when the last holder unregisters.
* test: fail if the surviving reader stops instead of keeps reading
Review caught that the iteration count alone proves nothing: had
LoopProcessLogData returned when the duplicate reader unregistered, the
counter would sit at 0 and the assertion would pass without a reader
ever having been there to spin. Check the reader is still running before
trusting its low count.
---------
Co-authored-by: Junker der Provinz <jdp@braethoria.com>
* rust volume: quick-repair redb on durable checkpoints
set_quick_repair(true) on the durable checkpoint transaction so an
OOM-killed volume server opens without a full-file repair scan.
* rust volume: optional redb insert_before bulk load
Behind redb-experimental-cursor (default off). Production binary
stays on sorted insert(). CI unit tests run both feature settings.
* rust volume: exercise insert_before across leaf splits
Replace the 5-key cfg clone with a 4000-key reverse-order rebuild
so CursorMut::insert_before hits page splits. CI runs the feature
only on storage::needle_map unit tests.
* rust volume: insert redb rebuilds in needle-id order
Unlink the .rdb before create (create does not truncate). Collapse
last-write-wins, then insert live keys sorted so 4.2.0 packs leaves.
* rust volume: rebuild redb from a BTreeMap and clear leftover keys
Peak rebuild memory is one ordered map instead of HashMap + Vec +
stable-sort scratch. Unlink stays best-effort: if it fails, retain
clears the leftover table before sorted insert. Compute idx metrics
before the write so a read error does not unlink a committed .rdb.
* rust volume: drop the extra redb read transaction on put/delete
put uses insert()'s previous value. delete gets then inserts the
tombstone in the same write transaction. Truncate the .idx row on
any failed redb write after the append.
* rust volume: unpack redb blobs through packed_to_needle_value
save_to_idx, ascending_visit, and collect_entries used the same
length-check copy as get. Route them through the helper so a
wrong-length value is absent everywhere, not a panic.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* rust volume: insert redb rebuilds in needle-id order
Unlink the .rdb before create (create does not truncate). Collapse
last-write-wins, then insert live keys sorted so 4.2.0 packs leaves.
* rust volume: rebuild redb from a BTreeMap and clear leftover keys
Peak rebuild memory is one ordered map instead of HashMap + Vec +
stable-sort scratch. Unlink stays best-effort: if it fails, retain
clears the leftover table before sorted insert. Compute idx metrics
before the write so a read error does not unlink a committed .rdb.
* shell: volume.check.disk — actionable verdict for diverged vacuumed replicas
When -resurrectMissingNeedles is gated off because both replicas have been
vacuumed (compaction revision > 0) — the normal state of any production
cluster — check.disk previously stopped at 'cannot prove they are missing
writes vs vacuumed deletes' and did nothing, leaving a diverged replica with
no repair path. volume.fix.replication does not catch it either: it only
acts when the replica COUNT is below the expected replication, never when
two replicas are both present but hold different live data.
Classify the divergence instead of dead-ending:
liveDivergence() counts live (non-deleted) needles present on one replica's
index but entirely absent from the other, in both directions. Tombstones
are excluded, so vacuum asymmetry (a compacted replica that dropped deleted
entries) is not mistaken for divergence.
reportDivergenceVerdict() turns the count into an operator action:
- one-sided (one replica has all the live data, the other has no unique
live needles) -> print the exact safe repair:
volume.copy -source <complete> -target <lagging> -volumeId <id>
Re-copying the complete replica is safe precisely because the lagging
side holds no unique live data; VolumeCopy's verify-before-destroy gate
independently confirms the source holds the volume before deleting the
target.
- two-sided (split-brain, both sides have unique live data) -> warn and
do NOT emit an auto repair; point to volume.fsck -findMissingChunksInFiler
to confirm the 'missing' needles are orphans before converging.
Report-only: no data is modified and the resurrection safety gate is
untouched. This is what lets a 13-volume diverged cluster be diagnosed and
repaired in minutes instead of by hand-diffing every index.
Observed motivating case: home SeaweedFS 4.45 cluster, 13 010 cross-rack
volumes diverged after failed replicate writes (ReplicatedWrite MaxAttempts=1
fire-and-forget), 11 one-sided + 2 two-sided, all repaired via volume.copy.
* shell: volume.check.disk — address review nits on divergence verdict
- Use pb.NewServerAddressFromDataNode (dialable ip:port, Address with Id
fallback) for the advertised volume.copy -source/-target instead of the
logical node Id, which may not be dialable.
- Make the one-sided verdict tombstone-aware: when the lagging replica has
been vacuumed, absent live needles may be valid deletions whose tombstones
were dropped, so a whole-volume re-copy would resurrect them. The command
is only advertised as safe when the lagging side is proven never-vacuumed
(compaction revision 0 read under -resurrectMissingNeedles); otherwise a
caveat is printed pointing at fsck/needle-level repair.
- Fix reversed copy direction when the source replica is the lagging one
(must copy complete -> lagging in both cases).
- Test: real tombstone (negative size) with the correct 0/0 expectation and
|| assertion; verdict test now covers dialable address, corrected
direction, and caveat on/off.
* shell: volume.check.disk — per-replica revision knowledge, no copy command for vacuumed lagging side
- Track srcRevKnown/tgtRevKnown separately: in unidirectional mode the
target revision IS read, so a proven never-vacuumed target no longer
gets a false resurrection warning (regression: bidi=false, target rev 0).
- A one-sided verdict now only emits the volume.copy command when the
lagging side is proven never-vacuumed; when it is vacuumed (or unproven)
the verdict refuses to print the destructive command and points at
fsck/needle-level repair instead — an appended caveat next to a ready-to-
paste copy command was still inviting the resurrection.
- deletionCaveat now returns the boolean safety decision.
* shell: volume.check.disk — preserve gRPC port, proven two-sided is not split-brain
- Emit the raw ServerAddress string (host:port.grpcPort) instead of
String()/ToHttpAddress(), which drops the custom gRPC port and would
make the suggested volume.copy dial the default port and fail.
- Two-sided divergence with both replicas proven never-vacuumed under
-resurrectMissingNeedles is mutually missed writes, not split-brain:
recommend re-running with -apply (in-place resurrection both
directions) instead of the split-brain no-auto-repair warning.
- Tests: case F (proven two-sided -> -apply, no split-brain warning),
case G (custom gRPC ports preserved in emitted addresses).
---------
Co-authored-by: timolow <tim@timolow.dev>
* rust worker: add install_default_crypto_provider helper
lance's aws backend pulls aws-lc-rs and reqwest's rustls-tls pulls ring,
so rustls 0.23 cannot auto-select a CryptoProvider and tonic's client TLS
panics on first use. Add install_default_crypto_provider, pinning the
default to aws-lc-rs, mirroring the Rust volume server's helper of the
same name. Includes a regression test that builds a TLS channel and
panics without the install in this crate, where both providers link.
* rust worker: install the crypto provider at startup
Call install_default_crypto_provider before any TLS use, the way the
Rust volume server does in its main. Without this a worker started with
--tls-ca/--tls-cert/--tls-key panics on the first admin dial (#11194).
* s3: gate IAM-cache gRPC RPCs behind admin Bearer auth
The SeaweedS3IamCacheServer registered on the S3 gateway's internal gRPC
port (default 0.0.0.0:18333) accepted PutIdentity/RemoveIdentity/PutPolicy/
DeletePolicy/GetPolicy/ListPolicies/PutGroup/RemoveGroup with no per-RPC
authentication. An unauthenticated network peer could call PutIdentity with
Actions:[Admin] and write straight into the live accessKeyIdent map that
the SigV4 path reads, bypassing S3 authentication entirely.
Mirror the filer's IamGrpcServer.checkAdminAuth: require a Bearer token
signed with jwt.filer_signing.key (read from the existing s3a.filerGuard)
at the top of every IAM-cache RPC. With no key configured the check is a
no-op, matching the rest of SeaweedFS's gRPC surface.
* credential: attach admin Bearer token to S3 IAM-cache propagation
The filer's PropagatingCredentialStore fans IAM mutations out to peer S3
servers over the SeaweedS3IamCache gRPC service. Now that the S3 handlers
require a Bearer token signed with jwt.filer_signing.key, attach one to the
outgoing propagation context (mirroring shell/iamAdminAuthContext). With no
key configured it is a no-op, so deployments that run without the signing
key keep working.
* credential: mint IAM-cache admin token after master discovery
propagateChange attached the admin Bearer token before ListClusterNodes,
so master-client retries could run down the (default 10s) token lifetime
before the peer S3 fan-out began, leaving peers to reject an expired token
and IAM caches stale. Move withIamCacheAdminAuth to after discovery
succeeds, immediately before the propagation timeout is derived.
* credential: cap IAM-cache propagation timeout below JWT lifetime
The propagation fan-out used a fixed 10s timeout. If an operator
configures jwt.filer_signing.expires_after_seconds below 10, the admin
token can expire while slower S3 peers are still being contacted, leaving
their IAM caches stale. Derive the propagation deadline as
min(10s, tokenTTL) so it never outlives the token. withIamCacheAdminAuth
now returns the token's lifetime (0 = no expiry) for this purpose.
Both tier-move handlers run in a detached tokio::spawn and report
progress through a closure that returns (), with the send result
discarded. Nothing observes the caller leaving, so an abandoned move
uploads or downloads the whole .dat anyway and then commits the
transition.
Go aborts both. Its progress callback returns `stream.Send`'s error,
which surfaces out of the reader in s3_upload.go:99 and the writer in
s3_download.go:84 and fails the transfer, so the volume info is never
rewritten. The Rust port dropped that by typing the callback as
FnMut(i64, f32) with no result.
Give the callback Go's signature -- FnMut(i64, f32) -> Result<(), String>
-- and abort when the caller's channel is closed. Checked on every part
rather than only where progress is reported, since the report is
rate-limited to one a second and would miss a caller that left in
between. A merely full channel is a slow reader, not a departed one, so
only TrySendError::Closed counts as cancellation.
Two consequences of aborting mid-transfer that the old code never had to
handle:
- upload_file now aborts the multipart upload when the transfer fails.
An abandoned multipart upload does not show up in an ordinary object
listing but still accrues storage charges until a lifecycle rule reaps
it, and cancellation makes that a routine path rather than a rare one.
- The tier-down handler removes the partial .dat. download_file
pre-allocates the destination to the object's full size, so an aborted
download leaves a file of the right length and the wrong content --
and this handler refuses to run at all when a local .dat exists, so
leaving one wedges every retry on "already on local disk" and a
restart would load the sparse file as the volume's data.
There is deliberately no check between a finished transfer and the
bookkeeping that follows. Once the object is in S3, or the .dat is on
disk, that bookkeeping is what makes the state consistent; stopping
there would leave an object paid for and referenced by nothing, or a
complete local .dat the volume still calls remote. Go does not gate
there either -- its callback only runs during the transfer.
Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* rust volume: stop a VolumeCopy whose caller has gone
VolumeCopy runs its copy in a detached tokio::spawn and reports progress
with the send error discarded, so nothing observes the client leaving.
When the caller cancels the RPC -- which weed-admin's batch balance does
routinely, starting far more copies than it finishes -- the server
streamed the whole volume from the source, wrote it to disk, and mounted
it. The destination is then left holding a volume nobody took delivery
of: its index cache is never reclaimed, and under replication=000 one
volume id ends up on two servers, both writable, which concurrent writes
can diverge.
Three checks now reach the task:
- Every chunk in copy_file_from_source, via the sender's is_closed().
This is the one that matters in practice. The first progress report is
128MB in, so for a smaller volume -- the ordinary balance move -- no
send ever happens and its result says nothing; only the closed channel
does. The sender is passed for the .idx and .vif copies too, with
reporting gated separately, so those phases notice as well.
- The throttle sleep, which for a throttled copy runs for seconds at a
time, now races the sender's closed() instead of being slept through.
- Immediately before mount_volume, and once at the top of the task.
Cancellation surfaces as an ordinary Err(Status::cancelled), so it lands
in the existing error branch that already removes the partial .dat/.idx/
.vif and the .note. That branch also logs now: the error otherwise went
to a channel nobody was reading, leaving the operator with the
balancer's "delete that copy, then re-run the move" and no cause.
This also clears the stranded read-only sources reported on the issue.
They are downstream of the orphan mount, not a separate defect:
LiveMoveVolume's cleanup probes the target before undoing the freeze
(volume_move.go:95, "the server can finish the copy and mount the target
even when the client loses the stream"), and when it finds a mounted
copy it cannot attribute, or cannot delete, it deliberately keeps the
source readonly rather than risk two writable replicas -- the messages
at volume_move.go:110 and :123. With nothing mounted on the target the
probe reports clean and the freeze is undone.
On Go parity: the progress send result is honoured here too, matching
`return false` in volume_grpc_copy.go. But that report is Go's only
abort signal, and measured against a 120MiB volume -- above the
throttler's activation threshold, below the 128MiB report interval -- a
Go destination mounts an abandoned copy as well. The issue's premise
that Go aborts holds only above the report interval. The Rust side now
stops in both cases; the Go behaviour is worth its own issue.
Tests: the integration test runs against both implementations and is
green on Go, red on Rust before this change. Its 192MiB fixture is sized
for two separate constraints, documented at the fixture: IoBytePerSecond
is a no-op below ~100ms of wall clock (64MiB copies in ~110ms on a tmpfs
loopback cluster), and the payload must exceed the 128MiB report
interval for the Go leg to pass at all. The two Rust unit tests cover
what the integration test cannot reach: cancellation detected with no
progress report at all, and the cleanup of the partial files plus the
.note.
Fixes#11186
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb
* rust volume: surface VolumeCopy cancellation as Status::cancelled
copy_file_from_source returned Result<_, String>, so the per-chunk
cancellation path -- the one that matters in practice for volumes
below the 128MB report interval -- was wrapped to Status::internal at
the call sites. The spawn logging branch then classified it as a
generic failure instead of the intended "abandoned by caller",
defeating the logging change in the same PR for the case that occurs
most often.
Return Result<_, Status> from copy_file_from_source: Status::cancelled
for caller-gone, Status::internal for the existing errors. Drop the
.map_err(|e| Status::internal(e)) at the three call sites.
The unit test now asserts the code is Cancelled, not just the message
text, so the classification is locked in.
* rust volume: close cancellation gaps in VolumeCopy
Address two review findings on the same PR:
1. Roll back a mount that races a departing caller. The pre-mount
is_closed() check cannot close the window between the check and
mount_volume: if the receiver drops in that gap, the volume mounts
and the final tx.send(Ok(...)) fails, but its error was discarded
(let _ =), so the task returned Ok(()) and the error branch never
ran. The destination then held an orphaned mounted replica — the
exact defect this PR prevents.
Fix: track a mounted flag. The final send now checks its result;
on failure it returns Status::cancelled, and the error branch
calls store.delete_volume (which unmounts AND removes the files)
when mounted is true, instead of only unlinking.
2. Observe cancellation while awaiting the source stream. The
per-chunk is_closed() check only runs after stream.message().await
returns. A stalled source (slow disk, partition, GC pause) never
delivers a chunk, so a caller that has already left cannot preempt
the read: the task, the source connection, and the partial files
(including the .note) all outlive the caller indefinitely.
Fix: race stream.message() against progress_tx.closed() in a
tokio::select!, so a departing caller preempts a stalled source.
Adds test_volume_copy_after_mount_cancellation_rolls_back_mount to
cover the after-mount rollback path. cargo test --release green
(497 + 5 + 1 + 28).
* rust volume: keep remote data on after-mount rollback, race RPC startup
Two review findings on the after-mount rollback added in ff51f6a:
1. The rollback called delete_volume(vid, false, false), i.e.
keep_remote_data=false. A remote-tier copy .vif points at the same
cloud object the source replica references, so destroying the
abandoned destination with keep_remote_data=false deletes the
source remote data via Volume::destroy backend.delete_file_blocking.
Use keep_remote_data=true, matching the pre-spawn delete_volume at
the top of volume_copy.
2. client.copy_file(copy_req).await (the initial RPC establishment)
was not raced against progress_tx.closed(). If the source stalls
before sending response headers, the per-message select! added in
ff51f6a is never reached, and the task, source connection, and
preallocated files outlive a departed caller. Race the RPC
establishment against progress_tx.closed() the same way.
cargo test --release green (497 + 5 + 1 + 28).
* rust volume: race master-configuration wait against caller cancellation
try_get_master_configuration().await was the last un-raced await in
the VolumeCopy task before copy_file_from_source. A stalled master
(or slow leader election) would hold the task and its .note past a
departing caller, since the per-chunk cancellation checks are never
reached. Race it against tx.closed() the same way the source stream
reads already are.
cargo test --release green (497 + 5 + 1 + 28).
* rust volume: fix after-mount rollback test to actually reach that path
The previous version of
test_volume_copy_after_mount_cancellation_rolls_back_mount dropped the
response immediately after volume_copy returned, so tx.is_closed() was
already true at the spawn first check and the task returned before
mount_volume. The test was green for the wrong reason: the mounted
flag, the rollback, and the keep_remote_data=true line were all
uncovered.
Use the store write lock as a seam: take it immediately after
volume_copy returns so the task runs the copy to completion with the
caller still attached, passes the pre-mount is_closed() check, then
parks entering the mount block. Drop the response (caller gone) and
release the guard: the task mounts, fails the final tx.send, and must
roll back via delete_volume.
Verified by setting the rollback guard to if false: the test fails
with "destination still holds a mounted volume". With the rollback
enabled, probe eprintlns confirmed the full path: about to mount ->
mounted = true -> final send failed -> rolling back mount -> rollback
done.
cargo test --release green (497 + 5 + 1 + 28).
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust volume: checkpoint the redb index durably every 1000 writes
Every put and delete on a redb-backed volume committed with
Durability::None and nothing ever committed durably, on the theory that
the .idx file is the source of truth. redb, however, keeps an entry in
its transaction tracker for every non-durable commit and cannot recycle
pages that were on disk at the last durable commit until a durable one
happens. With no durable commit for the life of the process, both grew
with every write, and .rdb files could bloat toward double size after a
restart (#11179, the hash-table rehash stacks in the memleak output).
The needle map now counts non-durable commits and reports when a
checkpoint is due; the volume takes it, data first: flush the .dat, then
the map fsyncs the .idx and commits redb durably, recording in the same
transaction how much of the .idx the table reflects. A checkpoint makes
the index durable, so the bytes it points at must be down before it, or
after a power loss the index would reference past the end of the .dat
and the volume would load read-only. A failed .dat flush skips the
checkpoint; it is retried on the next write.
Volume::close() now closes the needle map instead of only syncing it,
and the redb map's close() takes the same checkpoint. Before, a clean
shutdown left the table durable (redb flushes on drop) but the recorded
.idx size stale at its load-time value, so the next load replayed every
entry written since load on top of the counters.
On load, the redb map's counters now come from the whole .idx history,
the way Go's LevelDB map rebuilds them (newest entry first, with a bloom
filter of seen keys), instead of from the table's final state. Both the
reuse and the full-rebuild path use it, so overwritten and deleted bytes
keep counting as garbage across restarts, and the incremental replay of
the .idx tail only touches the table, which makes it idempotent whether
or not the table is ahead of the recorded .idx size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK
* rust volume: skip redundant .idx fsync on checkpoint after flush_idx
On the fsync=true write path, flush_idx() already fsyncs the .idx before
maybe_checkpoint_index() runs, so the checkpoint's own sync() fsyncs the
same file a second time for nothing. Thread an idx_already_synced flag
from the volume through maybe_checkpoint_index into checkpoint(sync_idx):
when it is true the checkpoint skips its .idx fsync and only does the
durable redb commit. The delete path and close() still sync (they have
not flushed the .idx beforehand).
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: saturate writes_since_checkpoint to prevent u32 overflow
If checkpoints keep failing (e.g. a persistent .dat flush failure whose
error is not EIO and so does not mark the volume read-only), the counter
increments on every write with no upper bound and wraps at ~4.3 billion.
Use saturating_add so it pins at u32::MAX instead, which keeps
checkpoint_due() true and retries on every subsequent write.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: only update max_file_key on live entries in idx metric rebuild
metrics_from_idx called maybe_set_max_file_key on every entry including
tombstones, but the live on_put path only calls it for puts and on_delete
never does. A tombstone always has a preceding put for the same key that
already set max_file_key, so the result is the same today; restricting it
to live entries makes the parity with the live path exact and self-evident.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: advance idx_file_offset only after redb commit succeeds
put() and delete() appended to the .idx file and advanced idx_file_offset
before committing to redb. If the redb commit failed, the offset included
the orphan row that redb doesn't reflect. A later checkpoint would record
that offset as "the table reflects up to here," and the reload would skip
the orphan row entirely — the entry becomes permanently unindexed.
Move the idx_file_offset increment to after the successful redb commit.
The .idx file still has the orphan row (append-only), but idx_file_offset
stays behind it, so the next checkpoint records the smaller offset and
the reload replays the orphan row back into redb.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: skip index checkpoint on close when .dat sync fails
Volume::close() discarded the .dat sync_all() result and always
checkpointed the redb index. If the .dat sync failed, the checkpoint
made the index durable with entries that may point past the unflushed
.dat tail, and a power loss would leave the volume read-only on reload
(the max_needle_end check fires).
Check the .dat sync result: on success, checkpoint as before; on
failure, call close_without_checkpoint() — sync the .idx and drop the
writer without a durable redb commit. META_IDX_SIZE stays at the last
successful checkpoint, so the reload replays the uncheckpointed tail
(redb still flushes on drop, but without recording idx_size).
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: schedule checkpoints on every index mutation path
maybe_checkpoint_index was only called from do_write_request and
do_delete_request. put_needle_index and write_needle_blob_and_index
also call NeedleMap::put, which increments writes_since_checkpoint,
but neither triggered the checkpoint. Through those paths the counter
could grow past the interval without ever being satisfied, leaving
non-durable redb transaction state until close().
Add maybe_checkpoint_index(false) after the successful nm.put in both
methods. The .dat flush inside maybe_checkpoint_index covers the blob
write in write_needle_blob_and_index; put_needle_index pairs with a
prior write_needle_blob, so the flush covers that too.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: truncate orphan .idx row on failed redb commit
Commit 947ee28 moved the idx_file_offset increment after the redb commit
so a failed commit doesn't advance the watermark. But the .idx file is
append-only: the orphan row stays in the file, and the next successful
write appends after it. That write's idx_file_offset += entry_size
advances past the orphan, so a later checkpoint records an offset that
makes the reload skip the orphan row — hiding a persisted put or
restoring a deleted needle.
On a failed redb commit, truncate the .idx file back to idx_file_offset
before returning the error. This removes the orphan row, so the next
write appends at the correct position and idx_file_offset stays a
contiguous replay watermark. Add a truncate_to method to IdxFileWriter
(set_len for std::fs::File) and a truncate_idx_to_offset helper.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: chrislusf <chris@chrislusf.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The star_history workflow referenced actions/setup-python@v8, which
does not exist, causing the workflow to fail at the "Set up job" step.
Pin to v7, matching the version used across the other workflows.
The Admin UI documents its read-only account as view-only and blocks its
write requests, but the authenticated read routes returned object-store
users with plaintext access and secret keys. A read-only admin user could
retrieve another user's live S3 credential pair from GET /api/users and
GET /api/users/{username} and use it directly against the S3 endpoint,
converting view-only access into the victim identity's object-store
authority.
Redact the reusable secret_key in GetUsers, GetUserDetails, and the
rendered users page whenever the requesting session has the read-only
role. The public access_key identifier is retained so identities remain
browsable; only the reusable secret is stripped. Admin and no-auth
sessions are unaffected.
* fix(master): honour -volume.fileSizeLimitMB on the master's /submit - #6748
`weed server -volume.fileSizeLimitMB=2048` still refused anything over
256MB, and the reason is not the one the report assumes: the option does
reach the volume server. The master does not use it. Uploads through the
master's /submit are buffered by submitForClientHandler, which passed a
hardcoded 256MB to needle.ParseUpload, so the master rejected what the
volume server it started would have accepted.
The limit is now passed in. `weed master` gains its own -fileSizeLimitMB
with the same 256 default, so a standalone master behaves exactly as
before, and `weed server` and `weed mini` hand it the value their volume
server already got.
* master.follower: take the same upload limit, and say which flag to match
Review found the follower left behind. It serves /submit like the leader
and buffers uploads under the same limit, but kept the fixed 256MB, so a
cluster raised above that would accept an upload through the leader and
refuse the identical one through a follower.
Two smaller points from the same review: the master's flag description
named only the standalone volume server's spelling, and now names the
weed server and weed mini form too; and the under-limit test asserted on
the error message alone, so it would have passed had the limit rejected
that payload with different wording. It now requires the request to get
past parsing.
* master: delete replica_placement_mismatch labels when volumes leave topology
Fixes#10804. Setting the gauge to 0 left stale Prometheus time series
that grew unbounded with volume churn; remove the label set on unregister
instead.
* master: delete replica_placement_mismatch only after last placement leaves
Unconditional DeleteLabelValues on UnRegisterVolumeLayout dropped the series
while other data nodes still held the volume, hiding under-replication until
the next collect cycle. Delete only when Lookup is empty, and cover the
two-copy case in a regression test.
admin: bind to loopback by default, refuse public unauthenticated bind
The admin HTTP server (port 23646) defaulted to binding 0.0.0.0 with
authentication disabled when -adminPassword was not supplied, exposing
the full admin REST API (user creation, credential issuance, bucket
deletion, filer deletion) unauthenticated on the network. This is the
footgun described in GHSA-m3m8-mrgq-hf9h.
Keep the no-auth mode for local dev, but remove the network exposure:
- Add -ip flag (default 127.0.0.1) so the server binds loopback only
unless the operator explicitly chooses a public address.
- Refuse to start when binding a non-loopback address with no
-adminPassword and no [https.admin] mTLS. The operator must enable
auth or use loopback.
- weed mini sets -ip from its existing -ip.bind; the guard does not
apply because mini calls startAdminServer directly, not runAdmin.
Addresses GHSA-m3m8-mrgq-hf9h.
* s3: warn when a lifecycle change leaves fast-path-stamped objects on their old TTL
The per-write TTL fast path (opt-in via s3.bucket.lifecycle.fastpath)
stamps a volume TTL at PutObject time that can't be taken back. When an
operator lengthens or removes an Expiration.Days rule (or deletes the
bucket lifecycle) on a fast-path-enabled bucket, objects already written
keep their baked-in TTL and won't be rescued by the change — unlike the
default worker-driven path, which re-evaluates the current rules each
pass. This is the data-loss direction described in #11183.
Surface it: Put/DeleteBucketLifecycle now emit a glog warning and set
X-Seaweed-Lifecycle-Fastpath-Warning on the response when the change
removes, disables, lengthens, or re-scopes a fast-path-eligible rule.
Shortening a rule does not warn (old objects simply expire later, not
data loss). Tag-only and overflow-day rules are never on the fast path
and never warn.
Addresses the warning half of option 2 in #11183.
* s3: address review — emit warning after mutation succeeds, fix ID-rename false positive
Two issues raised by CodeRabbit, Greptile, and Devin reviews:
1. Failed mutations retained the warning header. The warning was set on
the ResponseWriter before storeBucketLifecycleConfiguration /
clearStoredBucketLifecycleConfiguration was called; if that failed,
the error response carried a warning for a change that was never
applied. Now the reason is computed before the mutation but the log
and header are emitted only after it succeeds.
2. Rule renames produced false "removed" warnings. fastpathRuleKey used
Rule.ID as the sole identity when present, so renaming a rule (same
prefix/size/days, different ID) treated the old rule as removed.
Replaced with two-pass matching: first by ID, then by fast-path
predicates (prefix + size). An ID-only rename with unchanged
predicates and days no longer warns. Greedy matching ensures each
new rule is consumed by at most one old rule.
Added regression tests: ID-only rename (no warn), rename + lengthen
(warn), rename + shorten (no warn).
* feat(s3): make RenameObject idempotent for a retried request - #10661
A rename that succeeds but whose response is lost leaves the client with
no safe move: retrying returned 404, because the source is already gone,
so a retry was indistinguishable from a rename that never happened.
The destination now carries what the rename that created it was, under
x-seaweedfs-rename-token: the client's token, the source key and the
time. A retry that names the same token and the same source and
destination is answered 200 without touching anything. The same token
sent for a different rename is refused with 409 rather than silently
answered, and a token older than 24 hours is treated as unrelated so a
key cannot answer for a request indefinitely.
Requests without the header behave exactly as before.
* s3: answer a reused rename token with 409, not 400
The PR promised Conflict and the code returned Bad Request. 400 tells a
client its request was malformed and invites it to give up; this request
is well formed and resending it unchanged will not help, because what it
collides with is a rename the same token already stands for.
The status code is now asserted in a test, since it is the part of this
behaviour a client actually acts on.
* Update weed/s3api/s3err/s3api_errors.go
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: fix rename token review notes
- ErrIdempotentParameterMismatch returns 409 Conflict, not 400. The
comment and TestRenameTokenReuseAnswersConflict both expect 409; the
code regressed to 400 in a later commit.
- stampRenameToken: clarify that markRenameToken mutates srcEntry in
place, so the token reaches the destination via the move regardless
of whether the UpdateEntry succeeds. The precondition only guards the
pre-move write, not the move itself.
- Extract the handler retry branch into retryRenameDecision and add
TestRetryRenameDecision, covering the source-still-exists fallthrough
that was previously reasoned about but not tested.
* s3: IdempotentParameterMismatch returns 400, matching AWS docs
The AWS S3 RenameObject API documentation specifies HTTP Status Code: 400
for IdempotencyParameterMismatch. Revert the previous 409 change and align
the comment and test with the documented behavior.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The Rust volume server opens one redb database per volume and built
each with redb's defaults, which give every database a 1 GiB page
cache (0.9 GiB read cache + 0.1 GiB write buffer). With hundreds of
volumes behind one disk the process-wide ceiling was volumes x 1 GiB:
memory grew in proportion to the pages traffic touched, never shrank
when traffic stopped, and hosts running many instances were OOM-killed
under bulk ingest. redb, redbMedium and redbLarge were also treated
identically, so the "memory~performance" tiers did nothing.
Size the cache per tier instead: 4, 8 and 16 MiB per volume, mirroring
the Go server's 3/6/12 MiB LevelDB block cache + write buffer. Thread
the budget through RedbNeedleMap::new/load_from_idx so every open path
(create, reuse, full rebuild) uses Database::builder().set_cache_size.
Fixes#11179
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
No source changes: the API surface the needle map uses (Database
create/open/builder, set_cache_size, set_durability, tables, iterators)
is unchanged and the on-disk format is still v3, so existing .rdb files
open as-is. The 4.0.0 breaking changes (Drop on AccessGuardMut, removal
of the Legacy type) do not touch this crate.
Relevant to the redb-backed index (#11179):
- 4.1.0: optimizes cache usage and memory usage; ~1.5x faster writes.
- 4.2.0: Durability::None commits ~2x faster; pages freed by a durable
transaction are reused by the very next one; a crash-recovery fix for
a crash during repair of an earlier crash.
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Change the matplotlib figure size from (10, 4) to (10, 6) so the
star history chart renders vertically longer in the README. The
regenerated note/star_history.svg reflects the new 5:3 aspect ratio
(720x432pt) instead of the previous flat 2.5:1 (720x288pt).
The "Mount and exercise" step left the weed.exe mini server and the final
WinFsp mount running when it exited. The next step's pwsh.exe then failed
with STATUS_DLL_INIT_FAILED (0xC0000142), failing a job whose actual test
step had passed. The same code passed on both the PR branch and the next
master run, so this was a transient launch failure — but it was caused by
an unclean environment and made fatal by a diagnostic step.
Tear down all weed.exe processes at the end of the test step so subsequent
steps launch into a clean environment, and mark the Logs step
continue-on-error so a diagnostic step can never fail the job on its own.
2026-09-05 00:10:40 -07:00
Chris LuGitHubdevin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* add a per-mount cache_wait_ms to the remote storage mount mapping
A read of an uncached remote-only object waits on a hardcoded size tier
before it can fall back to the origin, so every ranged read of a large
remote-only object pays that wait. Carry the wait in the mount mapping so
it can be tuned, or set to zero, per mount.
* resolve the cache wait of an uncached remote-only read from its mount
The wait came only from the object size, so an operator could not trade
cache hits for time to first byte. Both read paths now resolve the mount
covering the object and let its cache_wait_ms replace the size tiers.
* read straight from the remote when a mount waits zero for its cache
A mount used as a streaming source pays the cache wait on every ranged
read of an object too large to finish caching, and the caching itself is
wasted work. A zero wait now skips the cache call, so both read paths go
to the origin immediately.
* let remote.mount set the cache wait of a mount
remote.mount -cacheWait=0 turns a mount into a streaming source, and any
other duration trades cache hits against time to first byte.
* keep the size based wait for a version-specific read
A read pinned to a version cannot fall back to the origin, since the
mounted remote only holds the current key, so a mount that opts out of
caching would leave it on the 503 retry loop forever.
* let the operator allow a remote-only read to dial an internal endpoint
The remote-mount read paths in the filer and the S3 gateway always refused
an endpoint resolving to a loopback or private host, so a mount backed by
an internal S3 could never be read from its origin, only through the local
cache. Both now take the allowance the volume server already has, still
off by default.
* skip the background cache of a mount that waits zero for its cache
GetObjectHandler kicks off caching for every remote-only read, so a mount
serving as a streaming source kept downloading whole objects even though no
read ever waited for them.
* cover a zero cache wait end to end
The read has to reach a real origin, so the harness also opts the filer and
the S3 gateway into dialing the loopback remote it already allows for the
volume server.
* resolve the S3 cache wait once so the background cache follows it too
The background cache that GetObjectHandler starts read the mount on its
own, so it skipped a version-specific read that the foreground path still
waits for. Both now ask the same resolver.
* answer 404 when the origin of a zero-wait read is gone
Metadata can outlive the object it points at, and with no cache to fill
the read would sit on the 503 retry path forever. The remote backends
already report a missing object as ErrRemoteObjectNotFound.
* open the origin at write time for a multipart range
Every part of a multipart Range is prepared before any is written, so
opening eagerly would hold one origin connection per part and leak the
ones already opened when a later part fails to open.
* reject a cache wait shorter than a millisecond
The mapping stores milliseconds, so -cacheWait=500us truncated to zero
and silently turned caching off instead of waiting.
* restore the doc comment of cacheRemoteObjectForStreamingWithShortTimeout
Extracting the wait resolver left its comment on the new function.
* stat the origin before committing a multipart range
Opening at write time keeps no connection through the preparation, but it
also moved a failure past the point where the multipart body picks the
response status, so a gone origin truncated a 206 instead of answering
404. One stat up front puts the status back.
* stat the origin once per request
Every part of a multipart Range is prepared on its own, so the preflight
ran once per range instead of once per read.
* map Azure and GCS stream not-found to ErrRemoteObjectNotFound
ReadFileAsStream on Azure and GCS returned provider-specific not-found
errors instead of ErrRemoteObjectNotFound, so a zero-wait read of a
deleted object was misclassified as a transient cache failure and
retried indefinitely. Map BlobNotFound and ErrObjectNotExist the same
way StatFile already does.
* Update weed/remote_storage/gcs/gcs_storage_client.go
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: count a TTL volume's age from its last write, not the .dat mtime
A delete appends a tombstone needle and vacuum rewrites the .dat wholesale,
so the file's mtime moves without any write ever landing. The loader read
lastModifiedTsSeconds back from that mtime, so every restart of a volume
taking delete traffic re-armed expired() for another full TTL: an
overwrite-heavy collection kept growing until it hit the max-volume cap.
Recover the clock from the newest .idx entry that is not a tombstone and
read that needle's append timestamp, falling back to the mtime when no
write is recoverable. Only TTL volumes pay for the scan.
Fixes#11160
* volume: count the .vif destroy time from the last write too
ExpireAtSec is what an EC volume is reclaimed on, and it was recomputed as
now+TTL every time the .vif was written. A read-only mark, a tier upload or
an EC encode therefore handed an already expiring volume another full TTL,
the same way the .dat mtime did.
Derive it from the volume's last write, falling back to now for a volume
that has not taken one yet so a fresh volume is not born expired.
* volume: mirror the last-write TTL clock in the Rust volume server
Same recovery as the Go loader: scan the .idx backwards for the newest
entry that is not a tombstone and take that needle's append timestamp,
leaving the clock on the .dat mtime when no write is recoverable.
* volume: mirror the last-write destroy time in the Rust volume server
Both .vif writers and the EC encode computed ExpireAtSec as now+TTL, the
same way Go did, so the destroy time moved every time the sidecar was
rewritten. Route all three through the volume's last write.
* volume: report the .dat mtime in the Rust heartbeat, like Go does
The Rust server reported its TTL clock as ModifiedAtSecond while Go
reports the .dat mtime. The shell's quiet-period gates (volume.tier.move,
volume.delete_empty) read that field as "last touched", which a delete
has to count towards even though the TTL clock deliberately ignores it --
and with the clock now recovered from the last write, the two drift
further apart.
* volume: take the newest write by timestamp on a vacuumed volume
The reverse .idx scan trusted position, which holds only while the .dat is
append ordered. Vacuum rewrites it in key order, and since an overwrite
keeps its original key, the highest-key survivor is not necessarily the
newest write -- the recovered clock could land up to a TTL early and take
the volume with data still inside its TTL.
A volume that has been vacuumed (CompactionRevision > 0) now takes the
maximum append timestamp over a bounded window of write entries instead.
An append-ordered volume still answers in one read.
* volume: never guess a vacuumed volume's last write, and resolve wrapped offsets
Two holes in the reverse scan, both from review:
A vacuumed volume's writes are ordered by key, so any of them can hold the
newest timestamp. Reading a capped window sampled the highest keys, which
could still miss a recently overwritten low-key needle and expire data
inside its TTL. The scan now covers every write a vacuumed volume indexes,
and a volume too large to scan keeps the .dat mtime rather than report a
partial maximum -- late is recoverable, early is not.
A .dat past MaxPossibleVolumeSize wraps the offsets in its .idx, so reading
a timestamp at the unwrapped offset picks up an unrelated needle. Resolve
the entry against the needle header first and retry one volume size in,
the way doCheckAndFixVolumeData already does.
* volume: drop GitHub issue references from TTL comments
* docs(readme): replace star-history.com with self-generated chart
The star-history.com SVG is a third-party dependency that can rate
limit or go down. Replace it with a GitHub Action that fetches
stargazers via the REST API and renders an SVG with matplotlib,
committing note/star_history.svg weekly. The README references the
committed file directly, so the chart has no runtime dependency on
any external service.
* ci(star-history): run daily instead of weekly
starchart.cc is rate-limiting the SVG endpoint, so the Stargazers
chart renders blank. Switch to star-history.com, which serves a live
SVG for this repo and links to the interactive chart.
The Patreon CTA and Gold Sponsors logos sat between the logo and the
project intro, pushing the actual description below the fold. Move the
whole block to a dedicated `# Sponsors #` section after `# License #`,
add it to the TOC, and give it a real markdown heading so the anchor
works on GitHub.
* mount: let volumeName take an explicit override
volumeName only ever derived the disk's label from -filer.path, -dir,
or the filer address, so a name that happened to collide with
something else - e.g. a UNC share's own name - could not be changed
without moving what was mounted. Give it an override parameter that
wins over all three; nothing passes one yet.
* mount: add -volumeName to name the disk explicitly
Windows has no equivalent of the "weed fuse" -o passthrough that lets
a Linux or macOS mount override its derived volname, so a name picked
up from -dir - e.g. a UNC share's own name - could not be changed
short of moving what was mounted. -volumeName overrides it on every
platform.
* mount: document -volumeName
* mount: scope -volumeName's help text to macOS and Windows
Linux has no volume-label mount option for -volumeName to feed, so
the flag's own description says where it applies instead of leaving
that unstated.
* mount: forward -volumeName through the weed fuse option parser
weed fuse (the /etc/fstab helper) turns -o key=value into the same
MountOptions weed mount takes, but volumeName had no case, so it fell
through to being forwarded as a literal, unrecognized FUSE option
instead of ever reaching mountOptions.volumeName.
* mount: apply -volumeName to FsName on Linux and FreeBSD
FsName only ever took the filer address and -filer.path, so
-volumeName had nothing to override there and silently did nothing;
the skipAutofs case still forces "fuse", since that name is what
util-linux/mount requires to recognize the pseudo filesystem.
* filer: guard the FoundationDB value size limit, not the transaction limit
An entry's whole chunk list is one FoundationDB value, and FDB caps a value at
100,000 bytes while a transaction may reach 10MB. UpdateEntry checked the
transaction limit, so every entry between the two limits passed the guard and
was rejected by FDB itself with error 2103 (Value length exceeds limit). The
failure surfaced inside the store rather than at the guard, so the S3 layer
dropped the connection and clients saw a network fault instead of an error.
Check the value limit in UpdateEntry and KvPut instead, after gzip and before
the transaction, with an error that names the limit it hit. The removed
transaction-size constant guarded nothing else: DeleteFolderChildren batches by
entry count.
Refs #11158
* filer: fold at 500 chunks in the foundationdb build
Manifest packing is what keeps a large file's entry small, but it only ran once
a flat chunk list reached 10000 chunks. A FoundationDB value stops at 100,000
bytes and an entry's whole chunk list is one value, which at ~100 bytes per
chunk record is about 1000 chunks -- so on FDB the write always failed before
packing could help: a 3.3 GiB PutObject at the default -maxMB=4 was already
past the limit.
FoundationDB support is its own build (`go build -tags foundationdb`, shipped
as its own image), so the batch is a build-time choice and needs no negotiation
at run time. The tagged build folds at 500, every other build keeps 10000 and
is untouched.
500 is not arbitrary: a single fold level leaves (chunks/batch) manifest
pointers plus up to (batch-1) unfolded chunks in the entry, so the reachable
chunk count is highest when the two terms are near equal. For a 100,000-byte
budget that optimum is 500, which holds an entry inside the limit up to
~250,000 chunks -- ~1 TB at -maxMB=4, against ~4 GB before. Larger files need
nested packing, which no batch size substitutes for.
One binary serves every role in that image, so the filer and each client that
folds -- S3, mount, WebDAV, weed shell, filer.copy -- agree on the batch by
construction. A binary built with the tag but pointed at another store folds
earlier than that store requires, costing one manifest blob per 500 chunks and
one read to resolve it.
Fixes#11158
* filer: fold with rollback inside MaybeManifestize, not beside it
A fold that fails midway has already uploaded manifest blobs for its earlier
batches, and returns only the data chunks -- dropping the manifests it had
separated out of the caller's list. Both were wrong in ways that mattered:
- AppendToEntry assigned that shortened list straight to entry.Chunks and
created the entry, so an append to an already-folded file whose fold
failed lost every previously folded chunk. weed mount had the same shape.
- cleanupChunks logged the error as "not good, but should be ok" and then
returned it through a named result, failing the whole CreateEntry or
UpdateEntry, while the blobs it had written stayed behind referenced by
nothing.
The S3 path was alone in handling this, through a private helper beside
MaybeManifestize. A second entry point next to the one everything else calls
just means the wrong one gets used, so the behaviour moves inside
MaybeManifestize: on failure it returns inputChunks as it received them, and
hands the blobs it saved to a deleteChunks callback. The filer, S3 and
filer.copy pass their existing deleters -- filer.copy already cleans up this
way after a failed upload -- and mount, WebDAV and weed shell pass nil, which
reports the blobs rather than collecting them, as before. Each caller keeps its
own error policy: the filer HTTP PUT path and filer.copy still fail the request,
the rest still continue with the flat list, which is a correct entry.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: HEAD with partNumber reports the part's size and range
HeadObject set its headers from the total object size and then only
validated the partNumber, so a client probing part 1 with HEAD got the
whole object's Content-Length and a 200 while the same GET returned the
part's size, a Content-Range and a 206.
Resolve the part's byte range before the headers are written, through the
range logic GetObject already used, and answer a partNumber HEAD as the
ranged HEAD that AWS documents.
* s3: answer an unsatisfiable partNumber with 416 InvalidPartNumber
GET and HEAD rejected a partNumber past the number of parts with 400
InvalidPart, the code for a missing part in CompleteMultipartUpload. AWS
answers a read of a part that does not exist with 416 InvalidPartNumber,
which lets a client probing for the part count tell the two apart.
The ceph suite pins RGW's 400 InvalidPart here, so the s3tests jobs patch
that expectation the way they already patch prefix ordering.
* s3: keep the whole-object checksum off a partNumber response
The stored checksum covers the whole object, so it is already withheld
from a ranged read. A partNumber HEAD now describes one part while the
request carries no Range header, so exclude it there too rather than
handing a client a checksum that does not match the bytes described.
* s3: resolve a partNumber against the parts the object records
Completion accepts ascending, not consecutive, part numbers, so the part
count is not the highest part number. Comparing the two rejected an
uploaded part 3 of a two-part object, and let a request for the absent
part 2 fall through to the positional chunk lookup and serve part 3's
bytes. Ask the recorded boundaries for the part instead, and keep the
count comparison for objects written before boundaries were stored.
* s3: apply a client Range within the part on HEAD too
GET narrowed the part by a Range sent alongside partNumber; HEAD reported
the whole part, so the two disagreed again for a request that carries
both. Move the narrowing into the shared range lookup so either verb
describes the same bytes.
* fix(chart): serve S3 internal gRPC with mTLS when security enabled
The security.toml generated by the chart has no [grpc.s3] section, so
security.LoadServerTLS(viper, "grpc.s3") returns nil in weed/command/s3.go
and the S3 server listens plaintext on its gRPC port (httpPort+10000 = 18333
by default). Workers dial that port with mTLS credentials (grpc.worker),
producing:
walker dispatch ...: rpc error: code = Unavailable desc = connection
error: desc = "transport: authentication handshake failed: tls: first
record does not look like a TLS handshake"
This breaks the s3_lifecycle worker's LifecycleDelete RPC path (recovery
walk, daily replay) and any S3->S3 IAM cache propagation would fail the
same way if clients enforced TLS.
Add [grpc.s3] reusing the client cert already mounted on s3 pods (or
s3.tlsSecret when set, mirroring the seaweedfs.s3.tlsArgs helper for the
HTTPS listener).
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* fix(chart): always use internal client cert for grpc.s3 identity
s3.tlsSecret is the public HTTPS listener certificate (possibly issued by
a public CA); internal gRPC peers only trust grpc.ca, so presenting it on
the internal gRPC port would break lifecycle/IAM RPC verification. Keep
the two trust domains separate.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.8-Flash-Next-ROCmFP4)
golang.org/x/image v0.44.0 is affected by CVE-2026-46603 (GO-2026-6222):
a denial of service via excessive memory allocation when decoding
malformed VP8L (lossless WebP) data. It is fixed in v0.45.0, released
2026-08-11.
The decoder is reachable from SeaweedFS: weed/images/resizing.go
blank-imports golang.org/x/image/webp, which registers the VP8L decoder
with image.Decode, so the filer image resizing path decodes attacker
supplied WebP data with the affected version.
This is a go.mod/go.sum only change produced by
`go get golang.org/x/image@v0.45.0 && go mod tidy`; no other dependency
moved. `go build ./weed/`, `go vet ./weed/images/...`,
`go test ./weed/images/...` and `go mod verify` all pass.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* filer: require a read token for the root listing
maybeCheckJwtAuthorization waved through every GET/HEAD on "/", so a filer
with jwt.filer_signing.read.key set still served its root directory listing --
entry names, sizes and chunks[].file_id -- to a caller holding no token at
all, and served the same listing to a token restricted by allowed_prefixes.
The exemption was added for health checks before the filer had /healthz and
/readyz. Both are registered on the default and read-only muxes ahead of the
"/" handler and answer without a token, so drop it.
Point the mTLS harness at /healthz, which is what it was probing for.
* filer: keep the jwt query parameter out of a proxied chunk request
The proxy stripped "jwt" from the forwarded query on reads only, on the
grounds that a writer's own credential travels there. It does not: an
uploader carries its AssignVolume token in the Authorization header, and the
query parameter on this path holds a filer credential.
Strip it for every method. A volume server has no business seeing a filer
token, and because security.GetJwt reads the query before the header,
relaying one would hide the writer's own token behind it.
* filer: dispatch the chunk proxy after the JWT gate
The ?proxyChunkId= branch returned before maybeCheckJwtAuthorization ran, so
GET, PUT, POST and DELETE against any needle in the cluster were reachable on
the filer's HTTP port with no filer credential, on a filer where every other
request answered 401. An anonymous caller read a stored object, replaced its
bytes, or deleted the needle, which the master's next vacuum makes permanent.
#10434 stopped the filer from minting a volume write token for that caller,
which closes the write half only where the volume server has a jwt.signing.key
of its own -- not the shipped default, and not what scaffold/security.toml
recommends for a filer deployment. The read half stayed open in every
configuration, because the filer mints the read token itself.
Move the dispatch below the gate. A file id carries no path, so a token
restricted by allowed_prefixes cannot be scoped against one and is refused
here; every consumer of this endpoint holds an unrestricted token.
* filer: mint the volume credential for a proxied write too
The proxy minted a volume token on reads and forwarded whatever the caller
sent on writes. #10434 made it that way because the branch ran ahead of the
JWT gate, so a token minted here would have been signed for an unauthenticated
caller; the branch now runs behind the gate, and the credential the caller
presents there is a filer one, which a volume server cannot validate and has
no business seeing.
Mint at the access level the request needs, and drop the caller's
Authorization when there is no key to mint from. A proxied uploader then needs
only the filer credential, instead of holding one for each hop with a single
header to put them in.
* mount, mq, filer.sync: send the filer credential for a proxied chunk
Every in-tree consumer of ?proxyChunkId= reached the filer anonymously: mount
and the broker put the AssignVolume token in the Authorization header, which
is a volume credential, and filer.sync sent nothing at all. That was enough
only while the branch ran ahead of the filer's JWT gate.
Build the URL through one helper, and pick the credential from the URL it
returns: a chunk proxied through a filer is a request to the filer, which
authorizes it and attaches the volume credential itself, so the token there is
a filer one at the access level the request needs.
* filer: honor -exposeDirectoryData
The flag was declared on all three commands that start a filer and read by
none of them: FilerOption.ExposeDirectoryData was only ever assigned from
filer.expose_directory_metadata in security.toml, so -exposeDirectoryData=false
silently left the listing exposed. Only the TOML key had any effect.
Plumb the flag through and let either switch turn the listing off.
* filer: count a proxied chunk request once
Moving the dispatch below the gate put it after the deferred request
observation, so every proxied chunk now landed in FilerRequestHistogram twice,
once under its HTTP method and once under chunkProxy. Name the deferred one
after the proxy instead, the way the unsupported-method branch already does,
which also gives the endpoint the status codes FilerRequestCounter records.
* filer: do not 404 a TUS session on a transient chunk-load failure
readTusSessionInfo already proved the session exists before
loadTusSessionChunks is called, so a failure there is a read failure,
not evidence the session is gone: a volume-server timeout or a
canceled request context surfaces through ListDirectoryEntries the
same way a missing session would.
Every such error was mapped to writeTusSessionNotFound, answering 404
to HEAD/PATCH and 204 to DELETE. A spec-compliant TUS client trusts
that and discards the session, orphaning every chunk it had committed
until the 24h expiry sweep, or forever if it never issues a DELETE.
Only an error matching filer_pb.ErrNotFound is now reported as not
found; anything else answers 500 so the client retries against the
same session instead of abandoning it.
* test: cover a TUS session's transient chunk-load failure
Adds a listErr hook to the in-memory test store, alongside the
existing commitErr/deleteErr, to simulate a store or RPC failure from
ListDirectoryEntries.
HEAD, PATCH and DELETE against a live session all answer with a
server error instead of a not-found status when the chunk listing
fails transiently, and the session is left on disk untouched. A
listing failure that genuinely means not found, filer_pb.ErrNotFound,
still answers 404 (204 for DELETE).
* filer.remote.sync: skip an upload whose source entry was deleted or rewritten
A replay from an earlier offset (-timeAgo) re-emits create and update
events for entries the filer has since deleted or rewritten. Their chunks
are gone from the volume servers, so the upload can never succeed, and
failing the event holds the sync offset before it: every restart of the
subscription replays it into the same dead chunks, and progress on
everything after it in the log is never persisted. One such entry stops
replication for the whole mount.
When the upload fails, look the entry up on the filer. Gone, or holding
other content than the event described, the event is superseded and is
skipped with an error log; the event that superseded it follows in the
log and brings the remote to the current state. Otherwise the failure
stands and the event is retried as before.
Fixes#11148
* filer.remote.sync: compare chunks by file id when deciding an event is superseded
filer.IsSameData compares chunk ETags, so a delete-and-recreate of
identical bytes, which stores the same content under new file ids and
drops the old ones, looked still as described and kept failing the event
on its dead chunks. Compare by file id with DoMinusChunks, the way the
filer itself decides which chunks an update leaves for deletion: the
event is superseded when the current entry no longer references every
chunk it named, and still as described when it does, including when more
chunks were appended after it.
* filer.remote.sync: ask the filer on the first failed upload attempt, not after the backoff
The superseded check ran after util.Retry had given up, so every dead
entry still cost the full retry cycle, about 13s, before it was skipped:
the SDK reports a missing chunk as "RequestError", which
IsTransientError takes as worth retrying. Move the check into the retry
loop with util.RetryOnError. Any failed attempt asks the filer, and the
loop stops at once when the entry is gone, surfacing errSuperseded for
the caller to skip. An entry the filer still holds keeps the retry policy
it had.
filer.remote.gateway shares retriedWriteFile and the same offset-pinning
processor, so its three call sites skip a superseded event the same way.
* filer: serve "//" paths at the cleaned path instead of redirecting
http.ServeMux redirects a non-canonical path ("//", "..") to its cleaned
form, but since Go 1.22 it builds the Location from the already-escaped
path, so it is percent-encoded twice (golang/go#79897). A client that
follows the redirect re-posts "/负极全景" as "/%25E8%25B4%259F...", and
the filer stores a directory literally named "%E8%B4%9F...".
Wrap the filer muxes in CleanPathHandler, which rewrites the request to
the same cleaned path ServeMux would have redirected to and dispatches
directly. The decoded name reaches the handler, the round trip goes
away, and clients that do not follow redirects work too.
Fixes#11125
* filer: keep RequestURI in step with the cleaned path
PostHandler derives storage rules, the bucket and the read-only check from
r.RequestURI while writing the entry at r.URL.Path. After CleanPathHandler
rewrote only the URL, a "//" or ".." request would be placed by the raw
path and written to the cleaned one. Rewrite RequestURI too, as the
redirect-following client used to.
* filer: match storage rules on the decoded write path
PostHandler resolved the storage rule from r.RequestURI, the raw
request-target. Clients percent-encode non-ASCII segments on the wire, so
a read-only or TTL rule configured on "/data/只读/" never matched a POST
to "/data/%E5%8F%AA%E8%AF%BB/" and the write went through. Use r.URL.Path,
the decoded path the entry is actually written to, as the header-based
destination check already does. The query string no longer reaches the
rule lookup, so the "?" trimming in the read-only error is gone.
2026-09-04 00:02:33 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: keep new volumes and writes off servers in maintenance mode
The master recorded a volume server's maintenance flag from the heartbeat
but never consulted it. A server in maintenance (#7977) is being drained,
yet the master kept creating volumes on it whenever it had free slots and
kept handing out its volumes for writes. Nothing on the volume server
blocks plain HTTP uploads either, so "read-only mode" was only a name.
Volume growth: a data node in maintenance mode reports zero free slots
through AvailableSpaceFor, which takes it out of every candidate list,
feasibility count and capacity reservation. Its slots still roll up into
its rack and data center, so the random offset drawn from those totals for
an other-rack or other-DC replica could land in space the walk then skips
and fail with "No free volume slot found!" while siblings had room; the
walk now folds the offset into the space that is actually eligible. This
also covers the pre-existing case of an over-committed sibling.
Assignment: a replica on a server in maintenance mode is treated like a
read-only replica in isAllWritable, so its volume leaves the writable
list and returns when the flag clears. Topology.SetDataNodeMaintenanceMode
re-evaluates the node's volumes on every change, since heartbeats are
digest-based and a full volume list may not follow for a long time. Reads
and lookups are untouched. The flag moves to an atomic so the assign and
growth paths can read it without the node lock.
Heartbeat: the Go volume server sent its state only when it changed, so a
master elected while a server sat in maintenance never learned about it.
The state now rides along on every heartbeat, as the Rust server already
does; the master's compare is an atomic swap, and only a change does work.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: hold maintenance mode through vacuum commit and mark-writable
SetVolumeAvailable and SetVolumeWritable put a volume back on the writable
list on the replica count alone. A vacuum that started before the server
entered maintenance, or a vacuum worker's mark-writable arriving after it,
handed the volume back to assignment with a replica on the draining server.
Heartbeats carry only changed volumes, so nothing re-evaluated it until the
volume itself changed.
Apply isAllWritable on both paths, the same test EnsureCorrectWritables
uses. Also pin that re-evaluating a volume a concurrent disconnect already
removed from its layout is a no-op.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: record a server's read-only notification on its node before judging the volume
A volume server notifies the master the moment it flips a volume between
read-only and writable, ahead of the heartbeat that repeats the flag. The
layout only set its per-location flag, so isAllWritable, which reads the
node's heartbeat copy, still saw the old value: a mark-writable was
withheld until the next heartbeat, and a re-evaluation landing between a
mark-readonly and its heartbeat put the volume back on the writable list.
Record the flag on the node's volume first. AddOrUpdateVolume keeps the
digest and the active volume count in step, so the heartbeat that follows
finds nothing to change.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: a read-only mark does not confirm a provisional volume
DataNode.SetVolumeReadOnly went through Disk.AddOrUpdateVolume, which
treats its input as a server report and so ended the grace period that
keeps a just-grown volume safe from a full report collected before the
grow. A volume marked read-only before its first report could then be
removed by that stale report.
Give Disk a SetVolumeReadOnly that flips the flag and keeps the digest and
active volume count in step without touching volumeAddedAt.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-03 23:50:12 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: confirm a missing RemoteEntry against the filer before re-uploading
The event is the entry as it was when the update was logged. A chmod or
utimes right after a write is logged while the sync is still uploading the
write, so it carries no RemoteEntry even though the object is on the remote
by the time it is processed. Gating on the event alone turned every such
update into a delete and a second upload of the same bytes; cp -p, rsync
and Django's FileSystemStorage all write that way.
Look up the filer's current entry when the event has no RemoteEntry: the
upload stamps it as soon as it completes, so the stamp is there for the
race and absent for a file that was never replicated. Skip the update when
the entry has since been deleted rather than upload from chunks that may be
gone; the delete event that follows removes the remote object.
Tests build entries from chunks, which is what IsSameData compares in
production, and cover both no-RemoteEntry cases through a stub filer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: do not delete the remote object before overwriting it in place
The update write path deleted the old object and then wrote the new one,
even when both are the same key. S3, GCS and Azure all overwrite on write,
so the delete bought nothing and left the remote with no object between
the two calls, or at all if the write then failed and pinned the offset.
On a versioned remote bucket it also left a delete marker per rewrite.
Delete only when the key changes, which is what the delete was for.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: trim comments
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-03 19:14:34 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Maintenance mode exists to fence a volume server so it can be evacuated
without taking new writes (#7977), but the gate added in #8115 also
rejected the RPCs evacuation issues against the source: VolumeMarkReadonly
(the first step of every move, and the failure reported in #11066),
VolumeDelete (the last step), and VolumeEcShardsDelete (the last step for
EC shards). volumeServer.evacuate, volume.move and ec.balance therefore
all failed on exactly the server they were meant to drain.
Those three RPCs only remove data or restrict the server further, the same
class as DeleteCollection and the unmount RPCs that were never gated, so
they are exempted from the maintenance check in both the Go and Rust
volume servers. Everything that adds data or reopens the server for
writes (AllocateVolume, WriteNeedleBlob, BatchDelete, VolumeCopy,
ReceiveFile, EC generate/copy/rebuild, vacuum, tiering, VolumeMarkWritable)
stays blocked. A side effect is that scrub can now fence broken volumes
readonly on a server already in maintenance.
Fixes#11066
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An entry whose content is rewritten unchanged before it first reached the
remote took the metadata-only branch, and UpdateFileMetadata returns early
when the extended attributes match without checking that the object is
there. shouldSendToRemote had already reported the entry as needing to be
sent, so the effect was that it stayed local for as long as its content did
not change, with the sync reporting healthy progress over it.
Require RemoteEntry to be set before treating an update as metadata-only.
Gating at the caller covers the S3, GCS and Azure clients, which share the
same early return.
Fixes#11139
2026-09-03 17:42:40 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Every image-signing job has failed since signing was added (#11129):
must provide --new-bundle-format or --bundle where applicable with
--signing-config or --use-signing-config
Cosign 3 turned on two defaults, not one. The action only disabled
--new-bundle-format to keep the .sig tag layout, but --use-signing-config
is still on, and cosign refuses that pairing because the signing-config
path has nowhere to write its verification material without a bundle.
Disabling it too falls back to the default Fulcio and Rekor URLs, the
same services the .sig layout always used. The verify step needs no
change: cosign verify looks for a referrer bundle first and falls back
to the .sig tag when there is none.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: make DeleteVolume errors inspectable with errors.Is
An absent volume wraps ErrVolumeNotFound and an only-empty refusal now
wraps ErrVolumeNotEmpty with %w instead of %v, so callers no longer have
to match on the message.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* volume server: return NotFound and FailedPrecondition from VolumeDelete
An absent volume maps to codes.NotFound and a non-empty volume under
only_empty to codes.FailedPrecondition, so a caller retiring a volume can
treat NotFound as already done. The store message is kept in the status
description because the EC empty-replica sweep still matches on it.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* wdclient: add LookupVolumeIdsAuthoritative
Bypasses the vid map and asks the provider directly, for callers where a
stale positive location is unsafe.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: add LookupDirectoryEntries batch lookup RPC
Up to 4096 exact-path lookups in one call, resolved concurrently with
results in request order, plus one deduplicated location lookup for every
volume the returned entries reference and per-fid read tokens when the
filer signs reads. unavailable_volume_is_miss lets cache-style callers
take an entry whose volume has no live location as a miss, resolved
against the master rather than the filer's location cache.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: test that an expired file entry is deleted on read
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: test that AssignVolume and CreateEntry resolve the same TTL rule
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* master: refuse partial lookups while warming up
LookupVolume returned Unavailable during warm-up only when every requested
volume was missing. A batch mixing a reported volume with one whose server
has not reconnected yet came back as a partial answer with a per-volume
not-found, which a caller treating the master as authoritative reads as
gone. Any not-found during warm-up is now Unavailable, which callers
already retry.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: build batch test requests instead of copying a proto message
Copying a generated message copies its internal mutex, which go vet's
copylocks check rejects.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: match ErrNotFound with errors.Is and state the miss rule's contract
A wrapped not-found from the store would otherwise be reported as an
error rather than a miss. The comments now say why a nil location map is
the only sign of an unanswered lookup: the provider returns nil when it
got no answer and a populated map, with unserved volumes reported as
errors, when the master did answer.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* volume server: map absent and non-empty VolumeDelete errors in the Rust server
Matches the Go server: an absent volume is NotFound and an only_empty
refusal is FailedPrecondition instead of Internal, with the messages the
EC empty-replica sweep matches on.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: test that a malformed entry keeps its error outside cache mode
Same test file as the enterprise tree, so the next sync sees one version.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
As an untyped constant it became int when passed to Infof, which
overflows on linux/386 and failed the 32-bit vet job. Every field it is
compared against is already uint64.
Claude-Session: https://claude.ai/code/session_015rYAmF8hV9yypb9yvy4A1z
* telemetry: tidy the server module after the protobuf bump
Claude-Session: https://claude.ai/code/session_01VGiphDxpsKMwu9XpUFhybQ
* telemetry: keep only clusters that store at least 10 GiB
Fresh weed server runs, CI jobs and throwaway containers each mint their
own cluster id. They came in at tens of thousands a day, were most of
the counted clusters and held almost none of the bytes, and the state
file and the metrics page grew with every one of them. Reports under
the floor are counted and dropped, and a state file written before the
floor sheds them on the first restart.
Claude-Session: https://claude.ai/code/session_01VGiphDxpsKMwu9XpUFhybQ
* master: report telemetry only once the cluster stores 10 GiB
A throwaway cluster no longer registers itself with its first report a
minute after start; a real one begins reporting at the first daily tick
after it crosses the floor.
Claude-Session: https://claude.ai/code/session_01VGiphDxpsKMwu9XpUFhybQ
* ci: composite action that signs and verifies an image with cosign
Keyless, by digest, with a verification pass against the calling workflow's
own identity right after signing. Signatures use the .sig tag layout rather
than the OCI-referrer bundle cosign 3 writes by default, since that is what
the verifiers people run today read. Dependabot is pointed at the action so
the cosign-installer pin keeps moving.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign every variant on both registries
The merge job signs each variant's multi-arch index on GHCR and Docker Hub
once the tag exists, recursively so the platform images are covered too.
latest re-tags the same manifest and inherits the signature.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker dev: sign the dev image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: sign a latest rebuilt by hand
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign the foundationdb image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: sign the per-version foundationdb and rocksdb builds
They push to the same repository as the releases, so an admission policy
that verifies chrislusf/seaweedfs would otherwise reject them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: document image signature verification
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin the actions the signing jobs newly run by commit
These run with registry credentials and the OIDC token that signs under
the repository's identity, so a retargeted tag upstream must not be able
to reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: pass the dispatch tag through env, not the script
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: complete Kyverno policy, digest note, identity scope
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: keep the dispatch tag out of the manifest script too
The step predates signing, but the job now holds the OIDC identity.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin every action in the jobs that sign
The jobs that hold the OIDC identity run these with registry credentials,
so a retargeted tag upstream must not reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: copy and sign the digest the run created, pin the rest
crane copy and the signature both resolved the tag, which another
publisher could move between the two steps. The index digest is read once,
right after it is created, and the Docker Hub copy and both signatures use
it. The manual latest rebuild gets the same treatment. The actions in these
jobs are pinned to commits, crane to v0.22.0 by checksum, and the sparse
checkout no longer keeps the token.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: the signing job checks out the workflow's own commit
The job only assembles and signs manifests, so nothing there needs the
source_ref checkout; the local signing action now comes from the same
revision as the workflow file that calls it.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: take the index digest from the create result
imagetools create writes the descriptor it pushed with --metadata-file
(buildx 0.32+, the runners ship 0.36), so the digest no longer comes from
re-resolving the tag even within the same step.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* http: try a volume server that failed to answer last
A cached location list is shuffled on every read, so once a replica dies
half the reads keep dialing it first and pay a connect failure or timeout
before the healthy replica answers. Remember, per host, when a request got
no answer at all and order such hosts last for the next half minute. Once
that passes, one read probes the host in its usual place while the others
keep it last until the probe settles, so a black-holed server costs one
stalled read per interval instead of one per read.
Nothing is ever skipped: a host that failed is still tried when the others
fail too. Any response, including an error status, counts as reachable.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* filer: refresh a chunk's locations after one of them fails
A mount's location cache is only relearned when every cached location
fails. When one replica dies and the other still answers, every read
succeeds and the dead replica stays in the cache, and in the shuffled
order it keeps being dialed first long after the master has dropped it.
When a read fails on one location and a later one answers, call the
refresh hook so the cached entry is dropped and looked up again. The read
that already paid for the failure returns its data; the reads after it
start from the locations the master knows now.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* http: claim the probe for every expired host, and try it first
The claim was only checked for the first url, so with two replicas whose
marks expired together the second was probed by every read at once. Claim
each expired host on its own and put the reads that won a claim ahead of
the reachable hosts, so a probe is always a real attempt and a lost claim
always means the host is tried last.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* filer: refresh a chunk's locations in the streaming read path too
The streaming loop had no refresh hook, so a manifest or streamed chunk
that failed on one cached location and was served by another kept the
stale entry until every location failed. Give it the same hook as the
buffered loop, built by one refreshUrls function shared by the reader
cache and the stream callers.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* http: probe at most one expired host per read
Claiming every expired host in one ordering left all but the first claim
without an attempt, since a read stops at its first answer, and a host that
had come back waited another interval for nothing. Claim only the first
expired host a read sees and leave the rest last and unclaimed, so each
following read probes one of them.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* test: start the live server before releasing the dead server's port
Closing the dead server first let the live server come up on the same
port, in which case the dead location answers and the partial failure
under test never happens.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
The cached-location test needs two files on one volume, but it wrote six
files into the six volumes a 001 layout starts with, and every so often
each file landed on its own volume and the test had nothing to probe.
Seven files leave no way to spread them out.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* helm: values-driven labels on every ingress
Each ingress already takes annotations from values, but its labels were
a fixed block, so tools that select ingresses by label (ExternalDNS
label filters, for one) had nothing to key on. Every ingress block now
has a labels map rendered after the standard app.kubernetes.io labels,
including the Traefik IngressRouteTCP that shares the filer gRPC values.
Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1
* helm ci: render check for ingress labels
Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1
weedfs_stream_mutate_error_test.go names its *streamMutateError local
sme, which codespell reads as a misspelling of same/some. It is an
identifier, so exempt it beside the other variable-name entries.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: rebuild a missing .idx from the .dat
Pointing -dir.idx at a directory that holds no index aborted the whole
volume server: checkIdxFile found no .idx and load() called glog.Fatalf.
Every row of the index is derivable from the .dat, so walk it in append
order and write the index back, which reproduces byte for byte what the
server's own writes had left in the old directory.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: keep the index co-located with the data in the Rust server
Go's load() drops back to the data directory when an .idx already sits
beside the .dat, so naming a --dir.idx does not strand a pre-existing
index. Rust had no such adjustment: it opened the new directory with
create, and the volume came up on an empty index with every needle
invisible.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: rebuild a missing .idx from the .dat in the Rust server
Mirrors the Go side. Rust did not abort on a missing index the way
checkIdxFile did; it opened the new directory with create and mounted the
volume on an empty index, so every needle read as missing while the .dat
still held the data. Walk the .dat in append order and write the index
back, byte for byte what the server's own writes had left behind.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a zero-padded .dat tail
An all-zero needle header is unwritten space, not a record. Go's .dat walk
keeps reading past it and would index a truncated data file's tail as
millions of needle 0 rows; the Rust walk already stops there. Stop the Go
rebuild at the same place.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: create the -dir.idx directory when it does not exist
Rust's DiskLocation creates the index directory as it takes it; Go only
resolved the path, so naming a directory that does not exist yet left every
volume unable to open or rebuild its index and took the server down.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a torn .dat record
A crash between writing a needle's header and its body leaves a record
whose declared size runs past the end of .dat. Indexing it puts a row in
the .idx that points at bytes that do not exist, which fails every read of
that needle and trips the past-EOF check on the next load. Stop at the
first record that does not fit, in both servers.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a negative-size header
A corrupt header whose size field is negative makes the .dat walk advance
backwards: NeedleBodyLength adds the negative size, so the next offset is
lower than the current one. The Go walk then reads at a negative offset and
the rebuild fails, which puts the volume server right back to exiting at
startup; the Rust walk seeks past EOF and truncates the index instead.
A negative size is never a record, so stop there.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: skip a volume whose index cannot be rebuilt, do not exit
glog.Fatalf calls os.Exit(255), so a rebuild that could not write -- a full
or read-only index directory -- put the server right back to dying at
startup for one bad volume. Return the error instead: loadExistingVolume
logs it and skips that volume, which is what the remote-volume branch just
above already does and what the Rust loader has always done.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: create the index directory from the rebuild too
The rebuild is the first thing to write into a fresh -dir.idx, and it runs
before the loaders that create the directory on their way to opening .idx.
Create it in both rebuilds so the ordering does not matter.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* ci: let codespell past the sme variable in the mount tests
weedfs_stream_mutate_error_test.go names its *streamMutateError local
sme, which codespell reads as a misspelling of same/some. It is an
identifier, so exempt it beside the other variable-name entries.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
-dir=/data fails on macOS, where the root filesystem is read-only, and
on any Linux box without root; -dir=./data is created on the spot.
go install of the weed package is refused because go.mod carries
replace directives, so the install script is the shortcut instead.
Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
The Helm values put filer metadata on a claim through filer.data, which
is what the chart reads; enablePVC was rendering a hostPath. Claims use
the cluster default storage class instead of local-path. The AWS CLI
test carries its own credentials, the compose download includes the
Prometheus config the compose file mounts, and the disk-read claim is
per blob, since large files are chunked.
Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
* filer: give the SQL stores' key-value reads their own connections
A listing holds the connection its rows are on for the whole iteration, and
FilerStoreWrapper calls maybeReadHardLink -> KvGet from inside that iteration,
so a hard-linked entry needs a second connection while the first is still busy.
Out of one bounded pool that is a deadlock: the listings fill the pool and then
wait for a connection none of them will release, and the wrapper's
context.WithoutCancel leaves the waiters without a deadline, so the filer stays
wedged rather than erroring.
The sqlite store shows it at its sharpest -- it allows a single connection, so
one listing over one hard-linked entry never returns. On postgres with
connection_max_open = 50, 60 concurrent listings over hard-linked entries made
no progress at all.
Key-value reads now run on their own pool, carved out of connection_max_open
rather than added to it, so the operator's cap still bounds what the store opens
against the database. An unbounded pool keeps a single pool: nothing can wait
there. sqlite's single connection becomes two, one per pool, and its writes get
a busy timeout so a write that meets the reader waits instead of failing.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* sqlite: keep both pools on one database, whatever the dbFile spells
A dbFile that already carries URI options got a second "?" appended, which the
driver reads as part of the preceding option value, and a bare :memory: is
private to each connection, so the key-value pool would open its own empty
database and every key-value operation would fail on a missing filemeta.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* sqlite: assert the busy timeout on the in-memory DSN too
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* rdma: drop the sidecar prototype
The Rust engine under it never touched a wire: rdma.rs fabricates pattern
bytes and the crate's default feature is mock-ucx, with real-ucx unimplemented
since the directory landed. Nothing builds it, no CI runs it, and its only
consumer is weed mount's RDMA client, removed next. Two 22MB binaries were
committed along with it.
Claude-Session: https://claude.ai/code/session_01X3zhqLYwQwEQCrRbuzQKvy
* mount: remove the RDMA client that spoke to the deleted sidecar
Its only server was the sidecar's HTTP API, and the path could never have
worked in production anyway: it served a single chunk per call, ignored the
buffer's chunk boundaries, and had no test. Removing it also removes the
per-handle cumulative-offset cache, which nothing else used.
The -rdma.* mount flags go with it. They defaulted to off and pointed at an
address no released build ever listened on.
Claude-Session: https://claude.ai/code/session_01X3zhqLYwQwEQCrRbuzQKvy
Lead with what SeaweedFS is and how to start it: one command, Docker,
Docker Compose, a production-shaped Helm values file, build from
source, scale out. Then why: fast, scalable, the S3 API surface with
operation counts, the lakehouse with S3 Tables and the engines that
share it, the cloud cache, and cross-cluster replication and the rest
of the feature list, each pointing at its wiki page.
The blob store walkthrough and the master and volume server internals
move to the Blob Store Architecture wiki page. The comparisons,
benchmark, enterprise and license sections stay. The dev plan is gone,
it was done.
Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
* shell: non-interactive mode exits non-zero when a command fails
A failed command in a piped weed shell run printed 'error: ...' but the process
still exited 0, so a CronJob wrapping e.g.
echo 's3.lifecycle.run-shard -shards 0-15' | weed shell -master=...
reported green while the run aborted partway (shards N+1..15 unwalked). An
unknown command likewise exited 0.
RunShell now returns the last command failure from the non-interactive stdin
path (unknown commands included), and the shell command exits 2 on it.
Interactive sessions are unchanged: errors are shown to the operator and the
session continues, exiting 0 as before.
* shell: route the piped-failure exit through main's shutdown path
Review follow-up: os.Exit(2) inside the shell command skipped main's shutdown
work. The command now records the status (SetCommandExitStatus) and returns
normally; main applies it via setExitStatus before exit(). exit() itself now
flushes sentry before os.Exit -- main's deferred sentry.Flush never ran on this
path (os.Exit skips defers), so the existing 'flush buffered events before the
program terminates' intent only worked for the autocomplete early-return.
Exit status 2 on a failed piped run is preserved (verified: piped success
exits 0, piped failing command exits 2).
* shell: test the registered-command failure path
Review follow-up: the error-propagation test only covered unknown commands.
A fake registered command now drives processEachCmd's real dispatch path:
a failing Do surfaces its exact error (errors.Is) and a succeeding one
returns nil. The non-interactive exit status itself is main-level plumbing,
verified end to end against the reproduction (piped failure exits 2).
* shell: trim the comments added with the exit status
Keep the non-obvious why -- why a piped run has to fail its wrapper, why the
status is recorded instead of os.Exit'ed -- and drop the narration.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* shell: fail a piped run with the status weed already uses for that
weed.go spends 1 on a command that failed and 2 on a usage or syntax error, and
runShell returns true precisely so the usage dump is skipped. Exiting 2 there
told a wrapper the command line was wrong.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
---------
Co-authored-by: Carlos Leyva <carlos.leyva@idener.es>
* filer: SQL store pool defaults survive concurrent walks (idle == open == 50, lifetime 300s)
The code defaults for the four SQL stores were connection_max_idle=2 with NO
default for connection_max_open (unlimited) or lifetime, while the scaffold
filer.toml documents 10/50/300 -- so an env-configured or minimal-toml filer got
the worst possible pool. Under a concurrent listing burst (s3.lifecycle.run-shard
walks 16 shards in parallel) every operation released above the 2 idle slots
closes its TCP connection, so the walk opens a fresh connection per operation
until the filer exhausts its ephemeral ports:
list /buckets/... : failed to connect ... dial tcp ...:5432:
connect: cannot assign requested address
Measured on a production filer: 0 -> 28k TIME_WAIT with only ~1.3k concurrent,
and in the minimal docker-compose reproduction (2000-dir bucket, port range
narrowed to 400): the whole range in TIME_WAIT with only ~12 ESTABLISHED.
Default all three knobs, with idle == open so released connections are kept and
reused: idle connections only accumulate up to the actual peak concurrency and
connection_max_lifetime_seconds recycles them, so a quiet deployment holds
nothing extra. An explicit 0 still disables the caps as before. The scaffold's
connection_max_idle moves 10 -> 50 to match.
With this change the same reproduction completes all 16 shards with the default
configuration (TIME_WAIT peak 19 vs the whole port range).
* filer: trim the SQL pool default comments
One line of the non-obvious why is enough; the rest narrated the code.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* filer: leave the SQL stores' connection_max_open unset
A listing holds its connection for the whole row iteration while its callback
runs another query -- FilerStoreWrapper.maybeReadHardLink does a KvGet per
hard-linked entry -- so every concurrent listing needs two connections from the
same pool. With a default cap, listings past the cap wedge: 60 concurrent
listings over hard-linked entries made no progress at all against a 50
connection pool, and the wrapper's context.WithoutCancel leaves the waiters
without a deadline.
The idle pool is what fixes the connection churn: idle 50 with an unbounded
max_open holds the same 14 postgres sessions across a 16-way listing burst that
opened 455 with idle 2.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
mount: name the disk after the mount point when the whole tree is mounted
The mounted path was the only thing that named the disk, so a mount of the
whole tree was labelled with the filer address and the only way to give it
a name was to mount a subtree under that name — which hides everything
outside it. Fall back to the mount point's own name first, so
-dir=\\seaweedfs\Images labels the disk while -filer.path stays "/".
Claude-Session: https://claude.ai/code/session_01Q9f8pWBXu1ceJvcQfYRQ7x
* fix(mount): make Mkdir exclusive so a concurrent duplicate fails with EEXIST
Mkdir sent CreateEntryRequest without OExcl, so the filer treated a
concurrent duplicate as an update and reported success to both callers;
the kernel's pre-mkdir lookup only masks this when the winner's create
is already visible. Set OExcl, map the entry-already-exists sentinel to
EEXIST instead of EIO, and drop the parent's children cache on the
losing side so the next lookup fetches the winner's entry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mount): route exclusive creates to the path's owner filer
The filer's per-path lock is filer-local and the store insert has
upsert semantics, so two mounts streaming to different filers can both
create the same path even with OExcl (measured 18/30 both-success on a
3-filer cluster). Hash the path over the sorted filer list so every
mount sends the same path's exclusive create to the same owner filer:
keep the mutation stream when it already targets the owner, fall back
to it when the owner is unreachable. Also let doUnary hand failed
creates to CreateEntry so the structured error code survives as EEXIST
instead of collapsing into the stream's generic EIO. Same race after
the change: 30/30 exactly one winner, every loser fails with EEXIST.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mount): review fixes — pin exclusive creates to the owner filer
An OExcl create now goes only to the path's owner filer: retrying on a
different filer would race the owner's possibly still-in-flight create
through a separate per-path lock, the very hole this routing closes. A
broken mutation stream retries the same owner over unary, and an
unreachable owner fails the create instead of degrading.
Pick the owner by rendezvous hashing so the choice is independent of the
configured filer order, and mounts configured with different but
overlapping lists still agree wherever the winning filer appears in
both.
Reject a stream create wrapper whose nested response is nil instead of
handing it to CreateEntry, which would dereference it.
Add ownerFilerAddress unit tests: order independence, subset agreement,
spread.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* mount: drop the client-side owner ring, the filer routes exclusive creates now
Exclusive creates are arbitrated cluster-wide on the server: the filer resolves
an OExcl create's ring owner and forwards one hop, so one filer's per-path lock
binds every creator — mount, S3, the HTTP surface and the Java client alike, not
only the ones that opted into a client-side ring.
That makes exclusiveCreateEntry redundant. It hashed the mount's configured
-filer list, which names a different owner than the master-maintained ring, and
failed the mkdir outright when its chosen owner was unreachable rather than
letting the ring reassign. Mkdir goes back to streamCreateEntry; OExcl and the
EEXIST mapping stay, and now mean what they say.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
* mount: cover the create error plumbing that turns a lost race into EEXIST
Letting a failed create's structured code survive doUnary is what makes a lost
mkdir race report EEXIST instead of EIO, and it had no test. Pull the two steps
out so they can be exercised without a live stream: hasCreateResponse decides
whether a response still carries a code to unwrap, createEntryFromResponse does
the unwrapping.
Reading the guard the other way round also says what it means — consume the
response only when there is no nested code left to recover — rather than
negating a type assertion inline.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
* mount: do not trust a create reply's shape before reading it
createEntryFromResponse read cr.ErrorCode without checking the nested response
was there. Nothing our filer sends is shaped that way, but the mount reads this
off the wire and a nil there panics the whole mount, so report it instead.
A top-level failure whose nested response carries no code was also returned as
success, silently losing the error. Fall back to the top-level errno when the
nested response explains nothing.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* lance: a maintenance job that sorts a table by its declared fields
Lance appends fragments in write order and has no notion of a sorted table, so
nothing but a rewrite establishes one, and nothing but another rewrite restores
it once rows have been appended. lance_sort reads the order from the dataset's
own configuration, falls back to the worker's, and rewrites the table in it.
The spec and the marker live in crates/sort rather than in the job, because
weed/worker/tasks/iceberg sorts too: two jobs that disagreed about what
"id desc nulls-first" means would be two features wearing one name.
The sort spills. lance builds its DataFusion runtime with a FairSpillPool and a
disk manager, but only when LanceExecutionOptions::use_spilling is set, and that
struct derives Default over a plain bool — so Scanner::try_into_stream, which
fills its options with ..Default::default(), is precisely the path that does not
spill. The job builds the plan with create_plan and executes it with spilling on
and the operator's memory budget.
The marker rides in the same commit as the data: Operation::Overwrite is the one
operation carrying config values alongside fragments, so a sorted table and the
record of its sorting cannot disagree. It records the version the sort read, not
the one it wrote, which is not knowable while the marker is being assembled.
Detection treats anything committed after the sort's own commit as data the sort
did not produce — row counts alone cannot see a rewrite that leaves the count
where it was, and such a table would look sorted forever.
Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y
* lance: identify a sorted table by the files it wrote, not its version
Review found three ways the version-based marker misjudges a table, and they
share a cause: the version a sort produces is not knowable while the marker is
being assembled, so the marker recorded the version it read and detection
inferred the rest. A commit that rebases past a conflict lands on a different
number, and the inference then reads a rewrite into an ordinary table — a full
re-sort, and its indices, for nothing.
Data file names do not have that problem. They are chosen before the commit, so
the commit can carry them, and they do not change with the version it lands on.
The marker now records how many files the sort wrote and a digest of their
names, and detection asks whether the table still holds them: the same files
means untouched, the same files followed by more means appended, anything else
means the data was replaced.
That also closes the hole the row threshold left. A replacement that grew the
table by fewer rows than min_unsorted_rows read as sorted, however many rows had
actually moved; the threshold now applies only where the sorted files are still
in place, which is what it was for. A marker without a row count is stale rather
than a zero to compare against, and deletes stop forcing a re-sort — they write
a deletion file beside the data rather than rewriting it, and removing rows does
not unsort the ones that remain.
Sort fields are also compared exactly rather than case-folded. Arrow schemas are
case-sensitive, so `id` and `ID` are two columns, and folding them together
rejected a valid order.
Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y
* lance: count the rows appended after a sort, not the table's net growth
Review found that rows deleted from the sorted fragments hide appended rows one
for one: the threshold compared the live row count against the count recorded at
sort time, so 800 deletions and 300 appends read as a table that shrank, and a
table where deletions keep pace with appends stays "sorted" with an unsorted
tail forever.
The fragments say it directly. The marker already records how many fragments the
sort wrote, so the ones after that prefix are exactly what arrived since, and
the manifest carries each fragment's live row count — physical rows less its
deletions. Counting those is the arithmetic the threshold was always meant to
do, and it needs no row count from the marker at all.
A fragment whose length the manifest does not record cannot be counted, and a
table that cannot be judged is one to sort rather than one to leave alone
forever, so an uncountable appended fragment reads as stale.
Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y
A sorted rewrite collected every row of a bin into one slice and sorted it
there, so a bin larger than the worker's heap could not be sorted at all.
sort_max_input_mb existed for that reason and skipped the bins it capped.
parquet-go's SortingWriter buffers sort_buffer_rows rows, encodes each buffer
as a sorted run, and merges the runs at close; backing those runs with a
FileBufferPool keeps them in files rather than on the heap. sort_spill_dir says
where, defaulting to the system temp directory — NewFileBufferPool resolves an
empty path to the working directory, which is not what an unset setting means.
The output now also declares its sorting columns, which the plain writer the
sorted path used never did.
Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y
* proto: resync the java copy of filer.proto
The Makefile keeps other/java/client/src/main/proto/filer.proto a verbatim copy,
but AssignVolumeResponse.fsync and SubscribeMetadataResponse.flushed_ts_ns
landed without it. Copy them over; no behaviour change.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
* filer: route exclusive and conditional creates to the entry's ring owner
CreateEntry with o_excl is a FindEntry-then-Insert. The per-path lock added for
it makes that atomic only on the filer running it, and the store's insert is an
upsert on every backend, so two filers both pass the existence check and both
report success. mkdir(2) then succeeds twice for the same path. The same hole
sits under the condition precondition, whose comment already told callers to
route the key's writes to the owner filer themselves.
Do it on the server instead, with the mechanism ObjectTransaction already uses:
resolve the entry's ring owner and forward one hop, bounded by is_moved. The
ring's membership comes from the master, so it tolerates a stale view and
reassigns when a filer dies, neither of which a client's configured filer list
can do. Every creator gets this — mount, S3, the filer's own HTTP surface, the
Java client — not only the ones that opted in.
Plain creates are upserts whoever applies them, so they stay local and pay
nothing. The route key shares the S3 gateway's namespace so an object's
ObjectTransaction and its CreateEntry land on the same filer's per-path lock.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
ObjectTransaction forwards to the ring owner so one filer's per-path lock
arbitrates every writer of a key. But a ring change hands the key over before
the new owner has rebuilt the locks the prior owner still holds, so for the
cooling-off window both can grant it. LockRing.PriorOwner exists for exactly
this and nothing consulted it.
Route to the prior owner while that window is open. LockRing.WriteOwner
resolves prior-else-current under one read lock, so the pair cannot come from
different rings and name the same filer twice.
An unreachable owner fails the request rather than falling back to the current
one. gRPC reports a response lost in transit as Unavailable, indistinguishable
from a request the owner never saw, so re-sending elsewhere could re-apply what
the owner already committed; and an owner unreachable from here may be
partitioned rather than down, still serving the key to everyone else — which is
the split brain the routing exists to prevent. The window is bounded: once it
closes the ring hands the key to its new owner.
The owner resolution and the forward move into writeOwner/forwardToWriteOwner
so the next routed RPC reuses them rather than copying the block.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
* filer: self-heal fetchWholeChunk on stale volume locations
Upstream #10156/#10800 wired cache invalidation into the buffer-based
read paths, but manifest resolution still goes through fetchWholeChunk,
which returns the raw error on failure. When cached volume locations
are stale (volume tiered to remote storage, server rolled), resolving
a large multipart file fails permanently even though other locations
are healthy.
Thread the ChunkGroup's cacheInvalidator through ResolveChunkManifest /
ResolveOneChunkManifest / fetchWholeChunk, and on failure invalidate,
re-lookup and retry once via the existing retryFetchWithFreshLocations
helper. The streaming bytesBuffer is reset before the retry so partial
bytes from the failed attempt cannot corrupt the manifest
proto.Unmarshal. Non-mount callers pass nil and keep their semantics.
* filer: move the manifest self-heal tests in with the other manifest tests
Also make the stale server stream a prefix and then abort mid-body, which is
what actually leaves partial bytes in the buffer: an HTTP error status returns
before ReadUrlAsStream ever calls the writer, so a 500 never exercised the
Reset the tests claimed to cover.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
* filer: keep the cached volume locations when a manifest read is cancelled
A cancelled or timed-out read says nothing about where the volume lives, so
dropping the location and going back to the master only costs the next reader
a round trip. PrepareStreamContentWithThrottler already guards its self-heal
this way. The guard also goes inside retryFetchWithFreshLocations, since the
caller can be cancelled between its own check and the invalidation, and that
covers the reader cache and prefetch paths too.
fetchWholeChunk returns the context error rather than the stream failure it
provoked, and ResolveOneChunkManifest wraps with %w so errors.Is still sees it.
That matters even where no invalidator is passed: volume.fsck resolves
manifests with nil and tells its own abort from a corrupt manifest that way,
so the cancellation check sits ahead of the nil-invalidator return.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
* filer: self-heal manifest reads on the filer and s3 paths too
Every caller that already holds the location cache backing its lookup function
can hand it over: the filer's read, copy and deletion paths and the log cache
have the MasterClient right there, and s3api has the FilerClient. MinusChunks
takes one for the same reason, since the deletion path resolves manifests
through it. Only the shell tools and the replication sinks, whose lookup
functions cache privately with nothing to invalidate, keep passing nil.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
---------
Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>
* shell: clean up the target copy when a merge upload fails
A replicated write commits the needle to the local volume before it fans out
to the other replicas, so an upload that reports failure can still have left a
copy on the target. fs.mergeVolumes printed "failed to move" and carried on,
so that copy stayed behind forever: the filer is never re-pointed at it, and
nothing else knows it exists.
One sick replica orphans roughly half the chunks of a merge, two thirds with
three copies, since the entry node is picked at random from the replicas and
the replica upload uses MaxAttempts 1. A volume with a single copy has no such
window: the write is one local append that either succeeds or leaves nothing.
Delete the needle we may have written before continuing. The source side
already did exactly this, so deleteMovedSourceNeedles is renamed to
deleteOrphanedNeedles and reused for both ends.
Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy
* shell: verify the cookie before deleting a merge target needle
The target cleanup deletes a needle an upload may or may not have written, and
BatchDelete matches on the needle id alone. Needle ids come from one global
sequence, so normally nothing else can hold that id — but a volume restored
from elsewhere, or one written either side of a master sequence reset, can, and
then a failed move deletes a live needle out from under its filer entry.
Have the volume server verify the cookie for those. Source needles keep
deleting by id: they are the ones the filer just pointed at, matching every
other filer-driven delete.
Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy
* shell: delete the target copies an abandoned manifest rewrite leaves
rewriteManifestChunk moves sub-chunks one at a time and only then uploads the
rewritten manifest. Every error after the first successful move — a nested
rewrite failing, the marshal, the manifest upload — returned without touching
the copies already written to the target volumes. The filer keeps pointing at
the old manifest, so those copies orphan, one per sub-chunk moved so far.
Track them alongside the sources and delete them on the way out. Nested
rewrites hand theirs up so an outer failure clears the whole subtree.
A failed UpdateEntry deliberately still leaks its copies: that error can also
mean the filer applied the update and lost the response, and deleting there
would turn a leak into data loss.
Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy
* shell: give the plan its room back when a manifest rewrite is abandoned
allocate reserves plannedSize against the chosen target for every move a
multi-target source makes, and release hands it back when the move fails.
Abandoning a manifest rewrite now deletes the copies that did land, so those
reservations stopped matching anything on disk: the plan kept counting bytes
that are gone and refused later chunks with "no target volume has room".
Release them alongside the delete. Nested rewrites hand theirs up so an outer
failure unwinds the whole subtree's accounting.
Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy
* iceberg: normalize foreign day partitions during manifest rewrite
* test: cover manifest rewrite with foreign day partitions
* iceberg: restore every foreign partition value, not just day transforms
iceberg-go takes a partition field's logical type from the last branch of
its Avro union, so a writer that spells an optional partition [<type>, null]
rather than [null, <type>] leaves the value as whatever the Avro decoder
produced. A day or date partition then arrives as a time.Time the manifest
writer cannot encode, and a time partition is worse: time.Duration converts
to int64 nanoseconds and silently records the wrong value.
ReadManifest sits next to ReadManifestList, the other shim for what foreign
writers put on the wire, and converts each partition value back to the
Iceberg representation for its field type.
Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR
* iceberg: read manifests that carry partition values through the shim
Compaction, delete rewrite and their detection passes read entries and write
the same partition values back into new manifests, so they fail on a foreign
day partition exactly as manifest rewrite does. Where filters see it too:
literalMatchesActual falls through to fmt.Sprint, so a time.Time renders as a
timestamp and never matches the day the user asked for.
The two remaining readers, orphan collection and the admin preview, only look
at file paths and stay on iceberg.ReadManifest.
Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR
* iceberg: convert partition values before the writer rebinds logical types
Dimonyga checked the manifests of a live Doris table: every input spells the
partition union null-first, with the date logical type present, so the union
ordering is not what breaks the merge.
The conversion is lazy. iceberg-go converts what the Avro decoder returned on
the first Partition() call, using the logical types read from the manifest
being parsed, and ManifestWriter.addEntry rebinds them to the manifest it is
about to write before it makes that call. A day partition is where the two
disagree -- iceberg-go's day transform reports an int32 result type, so the
manifest it writes carries no date logical type at all -- and an entry nobody
looked at in between converts against that and keeps its time.Time.
That is why only rewrite_manifests failed: compaction and delete rewrite group
entries by partitionKey(df.Partition()) first, which converts them, and a where
filter does the same. Reading every entry's partition here converts them all
while the manifest's own logical types are still in place.
Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* expose whether a volume replica is backed by remote storage
Volume locations returned by lookups do not indicate whether a replica
has been tiered to remote storage. Readers cannot distinguish a local
replica from a remote-backed one, so they may hit a remote-backed
replica first even when a local replica is available.
Add DataInRemote to the lookup location message, populate it from the
master's volume info, and carry it through the wdclient vid map so
clients can prefer local replicas when resolving chunk locations.
* wdclient: prefer local volume replicas over remote-tier replicas on lookup
LookupFileIdWithFallback (and the publicUrl variant in FilerClient)
didn't honor the DataInRemote flag when shuffling URLs, so the
DataInRemote patch only took effect in LookupVolumeServerUrl. Apply
the same ReorderToFront(localUrls) to sameDcUrls/otherDcUrls so
non-remote replicas stay at the front, matching the existing vidMap
convention.
* wdclient: propagate DataInRemote across tier transitions on existing replicas
When a volume is tiered to remote storage or a remote-backed replica is
restored locally, the cached DataInRemote on the same volume-server URL
stayed at its old value because two pieces of state never updated:
* master_grpc_server.go only split newVolumes and (already-tracked) volumes
into NewVids vs RemoteVids. ChangedVolumes went straight to NewVids, so
the broadcast announced the re-classified volume as a fresh arrival and
the client had no way to tell whether its existing cache was stale.
* vid_map.addLocationToMap early-returned when an entry already had the
same URL. A tier transition reports the same URL with DataInRemote
flipped, so the cached entry stayed at the old classification.
Wire both sides together: ChangedVolumes now go through the same IsRemote
split as newVolumes, and addLocationToMap replaces the existing entry in
place when the URL matches but DataInRemote has changed. The server
reference key only depends on URL/grpc port, so the refcount does not
move across the flip.
Adds vid_map_remote_transition_test.go covering the local->remote and
remote->local paths so the in-place update and the cache-key stability
are pinned by tests.
* wdclient: prefer local replicas across data-center boundaries
The previous local-first ordering hoisted local URLs to the front of each
data-center bucket separately, then concatenated same-DC before other-DC.
That meant a same-DC remote replica could still be tried before an
other-DC local replica even though the local one would answer cheaply.
Reorder once across the full candidate list: concatenate same-DC and
other-DC first, then ReorderToFront pulls every local replica to the very
front while preserving the DC preference inside each tier. Apply the same
ordering in all four lookup paths so the cached vidMap, the
LookupFileIdWithFallback provider path, FilerClient.GetLookupFileIdFunction
(PublicUrl-preferred variant), and the deprecated filer.LookupFn all agree:
- weed/wdclient/vid_map.go (LookupVolumeServerUrl)
- weed/wdclient/vidmap_client.go (LookupFileIdWithFallback)
- weed/wdclient/filer_client.go (LookupFileId)
- weed/filer/reader_at.go (LookupFn)
Strengthen the existing local-first tests: vidmap_client_localfirst_test
now asserts both endpoints are present (not just the local one is first),
and slice_test asserts an exact match instead of accepting two orderings.
Add TestLookupFileIdWithFallbackGlobalLocalFirst to pin the cross-DC
ordering invariant: any local replica (same or other DC) precedes every
remote-tier replica; within each tier DC1 precedes DC2.
Add docstrings to ToVolumeLocations, ReorderToFront, LookupVolumeServerUrl,
LookupFileId, GetVidLocations, GetLocations, LookupFileIdWithFallback, and
updateVidMap so the touched lookup paths are described in one place.
* topology: broadcast tier transitions on existing replicas
When a volume replica is tiered to remote storage or restored locally, the
wdclient's cached DataInRemote went stale: every connected client kept
preferring a remote-backed replica over a freshly restored local one, or
demoted a freshly tiered remote replica. The fix in commit 116982595 routed
ChangedVolumes to NewVids/RemoteVids on the master, but ApplyVolumeChanges
returned only fresh arrivals and previously servable replicas. An existing
replica whose IsRemote() classification flipped was neither, so it never
reached the broadcast loop and the wdclient never learned.
Make Disk.doAddOrUpdateVolume return a third signal -- tierTransition --
true exactly when an existing replica's IsRemote() flips. ApplyVolumeChanges
treats that as an arrival so the existing SendHeartbeat routing loop now
sees it. Add a master-side end-to-end test covering local->remote,
remote->local, no-op re-reports, and a mixed heartbeat that only announces
the tier transition.
Also add docstrings to LookupFileId, wdclientLocationsToPb, and
LookupVolume where the prior change touched their bodies.
* topology: broadcast tier transitions received through full reconciliation
The previous commit added tier-transition routing on the ChangedVolumes
delta path, but that is not the only way a re-tiered replica reaches the
master. After a digest mismatch the volume server resends a full Volumes
list, and SyncDataNodeRegistration applies the new IsRemote() classification
silently -- the changedVolumes return value was being thrown away. The
master therefore never broadcast NewVids/RemoteVids, and a wdclient connected
during the recovery kept the stale DataInRemote until it lost contact with
the master.
Surface the changed set through UpdateVolumes.changedVolumes (now covering
both ReadOnly flips and tier flips) and SyncDataNodeRegistration, then route
it through NewVids/RemoteVids in SendHeartbeat the same way the delta path
already does. Add an end-to-end test for the full reconciliation path.
* master: keep an EC volume's locations in the volume lookup
The nodes that answer for an EC volume hold shards, not a volume record,
so asking them for one fails. Dropping the location on that failure
emptied the result and turned every EC read through the master's HTTP
lookup and fid redirect into a 404.
Treat an absent volume record as a local read and keep the node in the
answer. The per-node conversion moves into topologyLocation so the EC
case is covered by a test.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* wdclient: replace a tier-flipped location without writing under a reader
GetLocations hands back the entry's own slice and the caller walks it
after the read lock is dropped, which is why every other mutation here
builds a new slice. Writing the flipped replica into the array in place
raced LookupVolumeServerUrl, reported by -race.
Copy the slice, swap the one element, and publish it.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* master: keep a remote volume on NewVids for older clients
Moving remote-tier volumes out of NewVids and into RemoteVids alone is a
wire break in the wrong direction. A master upgraded ahead of its filers
and mounts -- the usual order -- announces a tiered volume only on a
field the older client ignores, so the volume drops out of that client's
vid map entirely and reads for it fail.
Announce every volume on NewVids and repeat the remote-tier subset on
RemoteVids, so a new client still learns the tier and an old one keeps
the location. The routing moves into announceVolume, which the heartbeat
paths and their tests now share instead of each restating it.
On the client, RemoteVids no longer needs a second write per volume: the
tier is settled before anything is added.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* topology: split the volume snapshot by tier without copying the records
ToVolumeLocations runs on every KeepConnected, so a filer or mount
connecting made the master allocate a full VolumeInfo per volume per node
just to read four bytes of id off each one. AppendVolumeIds exists to
avoid exactly that.
Extend it to fill the remote-tier list alongside the full one, and use it
again in the snapshot.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* wdclient: keep the data-center preference ahead of the local-first ordering
Hoisting every local replica to the very front puts an other-DC local
read ahead of a same-DC remote one. When the remote tier sits in the same
region as the replicas -- the common arrangement -- that trades an
in-region GET for a WAN round trip and costs more than the remote read it
avoids.
Reorder inside each data-center bucket instead, so local still wins among
equals and the data-center preference still wins overall.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* operation: pick the read replica from one list
The local-preferring lookup built a list of local URLs and then branched
on whether it was empty, duplicating the random pick. Fall back by
filling the same list with every replica instead.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
---------
Co-authored-by: Bruce Zou <gift_secondst@msn.com>
Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>
The comment said enableReplication overrides "master & filer
defaultReplicaPlacement", but the two components take different keys:
master.defaultReplication and filer.defaultReplicaPlacement.
Claude-Session: https://claude.ai/code/session_014Yr1Asxq3qTjJo2r9G4cLX
* s3: ListParts on a completed or unknown upload answers NoSuchUpload
complete/abort delete the .uploads/<id> directory, but most filer stores list
a missing directory as empty rather than erroring, so listObjectParts answered
200 with an empty Parts list for an upload that no longer exists -- the same
response an open upload with no parts yet gets. AWS (and Ceph/RGW, MinIO)
answer NoSuchUpload, and clients lean on that: tusd derives the resumable
upload offset from the ListParts part sizes, so every completed upload read
back as zero bytes received.
Probe the upload record before listing, the way completeMultipartUpload
already does: not found, or a directory a late part write resurrected without
the destination key, answers NoSuchUpload. An open upload with no parts keeps
answering 200 with an empty list.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GGQEfVUvoATtwRCR8oC2jG
* s3: have the ListParts test filer refuse a directory it was not asked about
The fake answered the upload lookup on the name alone and the part listing
regardless of directory, so a wrong genUploadsFolder or upload-id suffix
would still have passed. Both calls now refuse any other directory with an
Internal error, which surfaces as ErrInternalError rather than the
NoSuchUpload the tests expect.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StsRz9wbu5dUCMGbFgPoRM
---------
Co-authored-by: tomislavcivcija <9787657+tomislavcivcija@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* s3: return the multipart object checksum in the CompleteMultipartUpload body
S3 carries the flexible-checksum members of CompleteMultipartUploadResult in
the XML body, not in response headers, so every SDK read back an empty
checksum from an upload that asked for one.
Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk
* s3: echo the checksum algorithm and type from CreateMultipartUpload
The upload directory already records both, but the response dropped them, so a
client could not confirm which checksum its parts had to carry.
Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk
* test: multipart upload reports the object checksum it was asked for
Covers every algorithm end to end: the create response echoes the algorithm and
type, the complete response carries the checksum, and it matches what a later
HEAD reports.
Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk
* master: survive a layout deleted twice
Two volume servers dropping the last replica of volumes that share a layout
both find it empty and both delete it. The loser's lookup misses, and the
single-value type assertion on the result crashed the master before the
caller could look at the found flag.
Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk
* master: remove a layout and read it back in one step
DeleteVolumeLayout looked the layout up and then deleted it, so two deleters
could each release the lookup ownership of the same layout, or one could find
nothing to release at all. Have the map hand back what it removed.
Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk
* s3tables: build the catalog attribute keys from one shared prefix
Every attribute the catalog stores on a bucket, namespace, table or view
entry is spelled out with the same literal prefix. Name it once in
s3_constants so code outside the package can recognize a catalog entry
without repeating the string.
Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1
* filer: keep empty folders that are s3tables catalog entries
A namespace, table or view is a directory whose extended attributes are
the catalog record. Its files can live elsewhere - a rename moves only
the catalog pointer and leaves the data at the old path, and a view has
no files at all - so an empty one is still a live entry.
Drop a table, then rename another table onto that name: the drop queues
the old table's folders, the rename recreates the name path, and two
minutes later the cleaner deletes it and cascades into the namespace,
losing a table the catalog still lists.
Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1
* filer: drop a queued cleanup when the folder is created again
A cleanup is queued against the folder that was found empty. If that
folder is deleted and a new one takes its name, the queue entry outlives
the folder it was about and the next pass deletes the replacement. A
drop followed by a rename onto the dropped name does exactly this: the
name path comes back as a live catalog entry two minutes before the
queue is read.
Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1
* filer_pb: build BFS child paths with FullPath.Child
A start path with a trailing slash produced "/dir//sub" for every
subdirectory, and the filer only trims a trailing slash, so those
listings came back empty and the walk stopped after the first level.
Claude-Session: https://claude.ai/code/session_01Jp9tXRpBv9gvh8fkaVFqxQ
* filer_pb: normalize the BFS start path
Entries directly under the start path were reported with the caller's
trailing slash, so filer.meta.backup wrote them under a directory the
incremental stream never names again.
Claude-Session: https://claude.ai/code/session_01Jp9tXRpBv9gvh8fkaVFqxQ
* s3: reject a version-id header that is not a valid path segment
putToFiler stored the client-supplied Seaweed-X-Amz-Version-Id header
verbatim into object metadata. That value is later read back and used
as a filer path component when building the .versions/v_<id> path, so a
value containing "/", "\" or ".." could steer retention/legal-hold
writes and remote-cache reads outside the object's own bucket tree.
Validate the header with isValidVersionID before storing it, the same
check the versioned read paths already apply, and reject the request
otherwise. Server-set version ids ("null" and generated hex) pass.
Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY
* s3: validate a stored version-id before using it as a path
The retention and legal-hold sinks build a .versions/v_<id> path from a
version id read back out of object metadata, and the remote-cache path
builder does the same from either the request or the stored id, without
the isValidVersionID check the other version-id consumers apply. Guard
these so a value that is not a valid path segment falls back to the
regular / unversioned path instead of steering the write or read out of
the bucket tree.
Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY
recursivelyCheckLocksWithClient tested EntryHasActiveLock only on
non-directory entries, so a directory-marker object (an S3 key ending
in "/") that carries retention or a legal hold was recursed into but
never lock-checked. DeleteBucket then saw no locks and removed the
bucket, destroying an object under active Object Lock along with the
rest of the bucket. DeleteObject already enforces the lock on the same
key, so the two paths disagreed.
Check the directory entry for an active lock before recursing.
Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY
ValidatePassword compared the stored and supplied passwords with
subtle.ConstantTimeCompare, which returns 1 for two zero-length slices.
A user provisioned for public-key-only auth has an empty stored password,
so an empty supplied password authenticated as that user whenever
"password" was among the enabled auth methods (the default).
Treat an empty stored or supplied password as a non-match.
Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY
mini hardcoded the per-directory volume limit to 0, so the volume server
always auto-sized it as free disk space divided by the volume size. That
sizing reserves a whole volume size for every writable volume, so a
workload spreading small objects over many buckets runs out of slots long
before the disk fills and assign starts failing with "no free volumes
left". Same name and semantics as the flag weed server already carries,
and still 0 (auto) by default.
Claude-Session: https://claude.ai/code/session_01BiQLeBvZLzG8XitjiypDKu
The replace pinned thrift to a post-v0.23.0 master commit so 32-bit
GOARCHes would compile. That fix shipped in v0.24.0, so the replace was
only overriding the v0.24.0 require back to the pseudo-version and
holding the build below the CVE-2026-43871 fix.
Claude-Session: https://claude.ai/code/session_01C5BpSeYD3yULWmfVXwPRmB
Membership was a linear scan over a slice, and ensureCorrectWritables runs it
for every volume on every heartbeat, so the master's steady-state cost per
volume server is quadratic in that server's volume count.
BenchmarkSyncDataNodeRegistration, median of 3:
1000 volumes 565.7us -> 535.0us -5.4%
100000 volumes 1.665s -> 55.3ms -96.7%
Allocations are unchanged at both sizes, so the difference is the scan.
Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1
The POSIX suites skirt this: pjdfstest's unlink/14.t covers only fstat and
pread on an unlinked descriptor — its driver has no fchmod at all and never
opens a directory. Pin the full rule in the FUSE integration suite:
ftruncate, fchmod, futimes, fstat, and the f*xattr calls keep working
between the removal of the last name and the final close, for a file after
unlink and a directory after rmdir, with nlink 0 and the changes visible to
a following fstat.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* mount: remember the entry of a directory removed while still referenced
A directory removed while a descriptor is open on it keeps its inode until
the kernel's final forget, but unlike a file it has no handle to live on
through: OpenDir hands out only a listing cursor. Keep the last-known entry
in memory, keyed by inode, from rmdir until that forget.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* mount: serve metadata ops on a removed open directory from its remembered entry
fchmod, futimens, and the f*xattr calls on a descriptor whose directory was
removed failed with ENOENT: maybeReadEntry resolved the inode to a path, and
rmdir had already dropped it. Fall back to the remembered entry the same way
an unlinked file falls back to its open handle. Mutations publish a changed
copy back rather than editing in place, so a concurrent reader never sees a
half-applied change, and the empty path keeps nlink 0 in every reply.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* mount: stash the entry the delete itself returned, not an earlier snapshot
A chmod landing between Rmdir's entry load and the delete RPC would be
resurrected pre-change: the remembered entry was the earlier local snapshot.
The filer serializes the delete against updates under the path lock and hands
the entry back in the delete event, so prefer that, keeping the local load
for the sticky-bit check and as fallback when no event comes back.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* mount: drop a remembered entry whose insert lost to the final forget
The forget's cleanup runs between RemovePath and the insert when the kernel
evicts the inode concurrently, finds nothing, and the entry would sit in the
map for the life of the mount. Re-check the inode after inserting and take
the entry back out; every interleaving now ends with the map empty.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* mount: insert the remembered entry under the inode table lock
The post-insert HasInode re-check could be fooled by inode number reuse: a
lookup landing between the forget and the check makes the number look alive
and the stale entry stays, keyed to someone else's inode. Do not check after
the fact — RemovePath now runs the retention callback inside its critical
section, where the forget that releases under the same lock cannot have run
and cannot be missed. Publishes need no such fence: their open descriptor
keeps the kernel from issuing the final forget in the first place.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* s3: share the per-key object authorization across copy and delete
AuthorizeCopySource and AuthorizeObjectDelete both authorize a key the request
URL does not name by evaluating the bucket policy and IAM against a synthetic
per-key request; only the method and action differed. Extract that into
authorizeObjectKeyAction and make the two callers thin wrappers. No behavior
change.
Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5
* s3: route POST Object uploads through the shared object authorization
POST Object uploads (presigned-POST / HTML form) authorized the write with only
the coarse per-identity Write action, unlike the other write paths which also
check the resolved object against the bucket policy and IAM. Route POST through
authorizeObjectKeyAction via a new AuthorizeObjectWrite so it is authorized like
the equivalent PUT.
Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5
* s3: test POST Object per-key authorization
Drives a signed POST upload and checks the per-key authorization decision for a
denied, permitted, and admin caller.
Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5
The version walk listed every directory with no prefix, transferring
all 1024-entry batches over gRPC and filtering gateway-side - and kept
paging past the point where names can no longer match. On wide
directories (many sibling orgs/jobs next to the requested prefix) that
is most of the transfer, decode, and CPU cost of every page.
Derive the next path component of the requested prefix per directory
level and hand it to the filer listing. A name holds no slash, so a
directory whose name does not start with the component cannot contain
a matching key and a file that does not cannot be one; stores with
native prefixed listing (sql, leveldb) turn this into a range scan and
stop the stream at the end of the prefix zone.
Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4
* volume: an already-deleted EC needle is not a delete failure
Deleting a needle that is already gone is what the caller asked for, and the
non-EC paths have always said so: BatchDelete reports StatusNotModified when
DeleteVolumeNeedle finds nothing to do, and DeleteHandler answers 404 from its
ReadVolumeNeedle pre-check. The EC branches had no such case, so ErrorDeleted
fell through to a generic failure -- 500 from both, and DeleteHandler also
counted it in VolumeServerFileWriteFailures, inflating a failure metric on a
replayed or duplicated delete.
The filer already tolerates this by string-matching "already deleted" on the
result, which leaves an error message load-bearing; the status is now right at
the source instead.
Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1
* volume: close the EC fixture's disk location
Close stops the location's disk-space goroutine and releases the mounted EC
volume's file handles, which otherwise live until the test binary exits.
Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1
* s3api: drop two redundant filer round-trips per listed version entry
ListObjectVersions paid two avoidable getEntry calls while walking a
bucket, both re-fetching data the walk already held:
- getObjectVersionList re-read the .versions directory entry that every
caller had just received from listing the parent directory (or from
its own sibling probe). Pass the entry down instead: one RPC saved
per object listed.
- getObjectOwnerFromVersion, on a version with no stamped owner,
re-fetched the same version entry its OwnerID had been extracted
from. The refetch cannot answer differently, so data written before
owners were stamped cost one futile RPC per listed version, forever.
All round-trips on this path are sequential, so on large versioned
buckets (Veeam-style workloads) they add up to a visible share of
per-page latency and gateway CPU.
Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4
* s3api: treat a nil .versions entry as an empty version list
filer_pb.GetEntry's contract permits (nil, nil) for an absent entry, and
the old internal lookup answered that case with an empty list. Keep that
answer now that the entry arrives from the caller.
Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4
The .versions lookup retried every error through the full backoff
ladder, so a missing key spent 12.7s (8 attempts, 100ms..6.4s) before
the pre-versioning fallback could answer. NotFound is an answer, not a
transient failure: gate the retries on isRetryableFilerErr, the same
classifier retryFilerOp already uses, which also stops retrying for
callers whose context is canceled or past its deadline.
GetObject already treats NotFound on .versions/ as definitive; this
brings the retention/tagging/ACL/attributes/delete/copy paths that go
through getLatestObjectVersion in line with it.
Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4
* filer.sync: count received, processed, and failed events and export in-flight jobs
The metadata processor admits at most -concurrency jobs and blocks the
subscription stream past that, so the backlog lives in the source filer's
metadata log and cannot be counted here. What can be measured honestly:
events read off the stream, replication outcomes, and worker saturation.
in_flight_jobs pinned at the concurrency limit means the sync itself is
the bottleneck; near zero means it is caught up or starved by the source.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: export replication lag in seconds
Lag is now minus the freshest of the processed watermark and the last
idle heartbeat: the watermark stops at the last real event, so a quiet
caught-up stream would otherwise show phantom lag. A ticker drives the
gauge because the offset callback only fires while events flow and
freezes exactly when the workers are saturated. Until the first event
or heartbeat the gauge stays unset rather than reporting lag against a
zero or stale resume offset.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: track replicated data sizes alongside event counts
An event count hides that 32 in-flight jobs can be 32 KB or 300 GB. Byte
counters mirror the event counters, and in_flight_bytes pairs with
in_flight_jobs. An event's size is the chunk delta - new chunks the old
entry does not already have - so deletes, renames, and attribute-only
updates count zero and byte rates reflect data movement, not metadata
churn.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* grafana: chart the new filer.sync metrics
The lag panel reads lag_seconds directly instead of deriving it from
sync_offset, and the sync row gains event rate, throughput, and the
in-flight jobs and bytes gauges.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: a pinned failure keeps showing as lag
An idle heartbeat means the stream is consumed, not that every event
replicated. While a permanent failure pins the watermark, letting the
heartbeat advance lag_seconds or the sync_offset gauge would report a
caught-up stream with an unreplicated event in it, so both now ignore
heartbeats until a restart replays the failure.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: in-flight gauges survive subscription retries
A subscription retry builds a new processor sharing the gauge children
while the old processor's jobs may still be draining, so setting the
gauge from either side's local count clobbers the other. Each job now
increments and decrements for itself, keeping the total truthful across
generations.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* master: keep the periodic growth scan to data centers hosting the layout
The rack-aware scan planned growth for every data center in the topology,
so a collection pinned to one DC (fs.configure -dataCenter) sprouted
volumes in all the others within one scan cycle. Plan only for data
centers already hosting the layout's volumes; an empty DC gets its
volumes from the DC-constrained assign that first asks for them. The
lastGrowCount divisor likewise counts only the racks the scan can plan
for.
Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5
* master: pin periodic must-grow growth to a single-DC layout's data center
The must-grow and crowded paths of the periodic loop grow with no
DataCenter, so even with the scan fixed a pinned collection's volumes
could still land in any DC once lastGrowCount demands more writables.
Stamp the grow request with the layout's data center when its volumes
all live in one; layouts spanning DCs keep unconstrained growth.
Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5
* master: never pin growth of a cross-DC-replicated layout
A layout whose replication spans data centers cannot legitimately live
in one DC; observing a single hosting DC there means the other DCs are
down. Do not encode that outage as a placement constraint.
Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5
* master: bound the hosting-DC walk by the answer it needs
listVolumeDataCenters walked every location of the layout under
accessLock — ~190ms for a million volumes, twice per layout per cycle,
stalling assigns behind the read lock. Stop once enough distinct DCs
answer the caller's question: two for the single-DC check, the
topology's DC count for the scan. A spanning million-volume layout now
finishes in microseconds; only a layout truly confined to fewer DCs
still pays a full walk, the same cost class as the under-replication
count this loop already takes each cycle.
Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5
Three dispatchers reuse one CommandEnv: the interactive shell, the master's
maintenance script runner, and the plugin worker's admin script handler. The
first two were fixed; this is the third.
It changes nothing today -- ForceNoLock already exempts this path -- so it is
here to keep the rule the same everywhere rather than resting on that exemption
staying in place.
* volume server: open volume files with O_NOATIME
Nothing reads the atime of .dat, .idx, .sdx, or EC files, but every
needle read still dirtied the inode: even relatime writes atime on the
first read after each write, so an actively written volume paid a
metadata write per read/write cycle, and strictatime mounts paid one
per read. Open the serving handles with O_NOATIME, falling back to a
plain open when the file belongs to another owner (EPERM).
Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD
* seaweed-volume: mirror the O_NOATIME volume file opens
Same change as the Go volume server: serving handles for .dat, .idx,
.sdx, .ecx, .ecj, and shard files open with O_NOATIME on Linux, with a
plain-open fallback on EPERM.
Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD
* route the tier-down and recreate .dat opens through the no-atime helper
Review caught the Rust tier-down swap opening the local .dat directly.
The Go swapToLocalDatBackend and the zero-length read-only .dat
recreate in maybeWriteSuperBlock had the same gap: all three install
long-lived serving handles.
Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD
* master: scope the startup capacity shed to a truly empty topology
The retryable "no volume server capacity registered yet" shed checked
capacity for the requested disk type, so a cluster serving only other
media -- where that capacity will never register -- shed every assign
until the client's deadline instead of failing fast. An unsteered write
to such a cluster hung for its full HTTP deadline and surfaced "context
deadline exceeded" in place of "No writable volumes". Shed only while
no disk type has any registered capacity, and name the unserved medium
in the fast failure.
Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R
* master: name the unserved medium for every fail-fast caller
The diagnostic sat in the growth-initiator block, so a follower joining
an in-flight growth and a growth-disabled master failed the same way
with only the generic pick error. Wrap at the fail-fast break instead,
which every caller reaches, and cover all three paths in the test.
Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R
noLock says "this invocation changes nothing" -- volume.balance, volume.move,
volume.copy, volume.merge and volume.fix.replication all set it for a dry run,
and none clears it. The CommandEnv is created once and reused by both
dispatchers, the interactive shell and the master's maintenance script runner,
so a simulation left every later command unlocked:
volume.balance -noLock # changes nothing
volume.move ... # mutates, and skips its lock
Reset before dispatch in both, where the invocation begins. forceNoLock is
untouched: that is set once, deliberately, for a trusted path.
* s3: resolve admin bucket subresources to their specific S3 actions
Encryption, requestPayment, publicAccessBlock and ownershipControls
requests reached the policy engines as s3:*, so only a policy granting
all of s3 could authorize them. Map each subresource to its AWS action,
with DELETE sharing the PUT permission as AWS does.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* s3: authorize CreateBucket as s3:CreateBucket in the policy engine
A plain bucket-level PUT is registered with ACTION_ADMIN, which resolved
to s3:*, so no attached policy short of s3:* could match it. Federated
sessions whose policy explicitly allowed s3:CreateBucket were always
denied while the same policy worked for object operations. Resolve it to
s3:CreateBucket, like DeleteBucket already resolves.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* filer_pb: walk a re-delivered directory only once in TraverseBfs
A directory handed back twice by a listing (a page-boundary race with
concurrent renames, or a store whose ordering misbehaves) was enqueued
twice; the second walk re-lists the same subtree and can keep the
traversal from ever terminating.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* filer_pb: fail a directory listing whose pagination stops advancing
A full page ending on the very name the cursor started from re-fetches
the same page forever; a store whose listing order does not advance past
the cursor turns any full-directory read into a silent infinite loop.
Return an error naming the stuck cursor instead.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* shell: skip foreign-collection manifests in fs.mergeVolumes
Every manifest chunk in the namespace was resolved, downloading its
manifest needle, even when the merge plan only touches one collection.
Sub-chunks live in the manifest's own collection, so a manifest on a
volume outside the plan's collections cannot reference a source volume;
skip it and spare a cluster-wide download pass that looks like a hang
after the real moves finish.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* sts: session duration no longer clamped to the web identity token exp
The assumed-role session lifetime is governed by DurationSeconds and the
configured tokenDuration/maxSessionLength, matching AWS. Clamping to the
already-verified token's exp made short-lived id_tokens (GitLab issues
~2-minute ones) yield unusable sessions regardless of configuration.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* sts: cover session duration against short-lived web identity tokens
The mock OIDC provider now carries the token exp through to the identity
like the real provider, so the integration test would catch the clamp.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* get volume topology info without volume details
Signed-off-by: lou <alex1988@outlook.com>
* master: rename VolumeListRequest.without_volumes to topology_only
The field shapes the reply rather than selecting volumes, and it leaves
out the ec shards too, which the old name denied. Match the message's
*_only style and say what a master that predates the field does with it.
Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5
* master: refuse topology_only combined with a volume selector
A topology_only request that also names a collection or volume ids
contradicts itself, and answering either half in silence surprises the
caller. Answer InvalidArgument from both VolumeList and its stream,
before the stream sends its header.
Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5
---------
Signed-off-by: lou <alex1988@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
mq: remove emptied topics after the cleanup iteration, not inside it
cleanupIdlePartitions called manager.topics.Remove from inside
manager.topics.IterCb. IterCb holds the shard's read lock while running
the callback, and Remove takes the same shard's write lock, so removing
an emptied topic self-deadlocked the cleanup goroutine. The pending
writer then blocked every later reader of that shard, permanently
hanging ListTopicsInMemory and, for shard-mates, TopicExistsInMemory.
On the Kafka gateway this surfaced as flaky e2e consumer-group tests:
one minute after any earlier topic went idle, the broker's first
'Removing empty topic' wedged the map, every gateway
ListTopics/TopicExists RPC burned its full 5s timeout, Metadata could no
longer finish inside kafka-go's 5s coordinator deadline, and consumer
groups looped in PreparingRebalance until the test timed out.
Collect the emptied topic keys during the iteration and remove them
afterwards via RemoveCb, re-checking emptiness under the shard lock so a
topic that just gained a partition is kept.
Claude-Session: https://claude.ai/code/session_014yA6c8JQcY6MqPXCT13yYA
An assign arriving before any volume server has heartbeated saw zero
available space and failed outright with a plain error no client retries,
so the first write to a fresh bucket answered 500 while the cluster was
still starting. Distinguish a topology with no registered capacity from a
genuinely full one: fail fast only when registered capacity is exhausted,
and shed ResourceExhausted otherwise so the client's retry budget rides
out the startup window.
Claude-Session: https://claude.ai/code/session_018G9kWFgy8BaBAEkYV3YL9n
* ec: refuse to mount a 0-byte shard file when the index has entries
The startup scan already skips (and eventually deletes) zero-sized shard
files as residue of a failed copy, but the mount RPC path opens the file
directly with no size check, so an explicit VolumeEcShardsMount over a
truncated file registers a size-0 claim. A registered empty shard serves
nothing while advertising ownership: with placement pinned to the owning
disk, it would keep attracting re-copies to a file that was never valid.
The one legitimate 0-byte shard is the empty volume's: encoding a volume
with no live needles produces a 0-byte .ecx and 0-byte shards, and that
mount must keep working (TestMountEcShards_EmptyEcxMountsSuccessfully).
So the gate compares against the index: AddEcVolumeShard (Go) and
EcVolume::add_shard (Rust) refuse a 0-byte shard file only when the
volume's .ecx has entries. Go's AddEcVolumeShard grows an error return
for this; the loader cleans up the refused shard and, when it just
created the EcVolume, unregisters that too. The mount loop already
collects non-ENOENT failures per disk and keeps scanning, so a sibling
disk holding a real copy still wins.
Regression tests in both trees: an empty shard beside an index with
entries is refused and leaves nothing registered; an empty shard of an
empty volume still mounts.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* ec: release the duplicate shard when a mount retry re-loads it
Review follow-up: AddEcVolumeShard keeps the existing shard and reports
added=false for a shard this disk already registered, but the loader
discarded that result, so every retried LoadEcShard leaked the duplicate
it had just opened — an fd and a mount-gauge increment per retry. Release
both and return the existing volume. Regression test pins the gauge.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* ec: close the test DiskLocation instead of only its EC volumes
Review follow-up: DiskLocation.Close() also stops the background
goroutine NewDiskLocation starts; closeEcVolumes left it running for the
rest of the test process. Both uses are this PR's own tests.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* rust: unregister the just-created EcVolume when its first mount is refused
Review follow-up: when the first mount of a volume rejects its shard
(e.g. the new 0-byte-beside-nonempty-index refusal), the Rust mount path
had already inserted the EcVolume and propagated the error without
removing it — a zero-shard registration advertising a mount that serves
no data while pinning the .ecx/.ecj descriptors (and, since placement's
mounted tier keys off it, steering shard placement at this disk). Remove
it on the way out, exactly as the Go loader already does; a volume that
already holds shards keeps them (the RPC's first-error-aborts contract).
Regression test covers both.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* rust: skip already mounted shards on a mount retry
Review follow-up: EcVolume::add_shard replaces self.shards[id] for a
shard the volume already holds, and the mount loop then bumps the
ec_shards gauge although the mounted count did not grow — gauge drift on
every mount retry, and a serving fd swapped for no reason. Skip shard
ids the volume already reports, mirroring Go's AddEcVolumeShard
added=false handling. Regression test pins the gauge across a duplicate
mount (unique collection label: the gauge is process-global and tests
run in parallel).
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* ec: pin auto-selected shard placement to the disk that already owns the shard
A multi-disk server legitimately mounts one EC volume on several disks, so
FindEcShardTargetLocation's per-volume tiers tie at "mounted" and the
free-shard-count tie-break decides — pointing at whichever disk is emptier,
not at the disk that already holds the shard being placed. A re-copy of a
shard the server already has (a retried ec.balance / ec.rebuild move) then
lands on a sibling disk, and both disks register the same (volume, shard id):
the shard is reported to the master from two disk ids, and which claimant
serves reads or survives a later unmount/delete becomes an accident of
Locations order.
Add a tier above "mounted": a disk that already claims one of the shard ids
being placed wins, ahead of the space filters too — re-copying in place
needs no new shard slot, and a genuinely full disk should fail the write
rather than silently split the claim. Applied to the Go selector and the
VolumeEcShardsCopy auto-select (ReceiveFile refuses mounted EC volumes, so
no claim can exist there) and mirrored in the Rust volume server.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* ec: refuse a copy batch whose shards are already owned by different disks
Review follow-up: ownership-aware selection ranks a mixed-owner batch
(shard 0 on disk A, shard 2 on disk B — the legitimate multi-disk spread)
into one destination, so the copy would still duplicate the losing disk's
claim. No production caller sends such a batch (balance moves one shard,
rebuild and encode copy shards the target lacks), so fail closed: report
every owning disk via Store.EcShardOwnerDisks and refuse the copy with an
error naming them, telling the caller to split per shard or pass disk_id.
Go and Rust, with unit tests for the owner-reporting contract.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
On 32-bit systems the generation fields becomes unaligned. Not an issue
on 64bit.
Discovered on 32bit raspbian, validated by manual patch (in addition to
test).
* ec: uniform shard block layout
An EC volume is striped as 1GiB blocks until less than one row remains, then
1MiB blocks, and consecutive blocks land on different shards. With ec.encode's
-fullPercent 95 against the 30GiB default limit, ~30% of every volume sits in
that 1MiB tail, so a 4MB filer chunk there is five stripes on five servers.
New encodes now use one block per shard, sized ceil(datSize/dataShards) rounded
up to 1MiB and recorded in the .vif (EcShardConfig.block_size, also carried by
the .ecsum manifest). A needle now maps to one shard unless it is larger than
the block or straddles a boundary. The chosen size equals the legacy layout's
padded shard length for every input, so shard sizes, capacity math, and the
shard-size credibility checks are unchanged; only the byte placement moved.
Reads, decode, and scrub resolve the block sizes from the volume's .vif;
absence keeps the legacy interpretation, so existing EC volumes read exactly as
before. Rebuild is layout-agnostic. weed fix -ecx recovers the layout from the
.vif, else the .ecsum sidecar, and with neither de-stripes under both candidate
layouts and keeps the one that indexes more valid needles.
Same change in the Rust volume server, which now also streams the encode in
256KB sub-batches like Go instead of allocating whole blocks, and computes the
large-row count as shardSize/largeBlock to match Go on exact multiples. On a
26MB fixture both encoders produce byte-identical shards, and a Go-written .vif
parses in Rust with the block size intact.
* ec: resolve the rust ecx rebuild through the recorded layout
The Rust rebuild path regenerated a lost .ecx by scanning the logical .dat
through a hand-rolled pure-1MiB striping, which was already wrong for legacy
volumes with large-block rows and is wrong for any uniform volume with a block
past 1MiB. Route the scan through locate_data with the .vif-recorded block
size, the same mapping the read path uses. Also seed the new tests' random
data instead of the deprecated global math/rand.Read.
* ec: fail the Rust ecx rebuild on any shard read error
A read error mid-scan published the entries collected so far as a
successful .ecx, and read_at's byte count was ignored so a legal short
read passed as complete — a truncated or failing shard could produce a
silently incomplete recovery index. Exact-read semantics in
read_from_data_shards, error propagation in the needle walk, and a
truncated-shard regression test.
* ec: fail the mount on an unreadable or malformed vif
Both servers silently fell back to the legacy layout when an existing
.vif could not be read or parsed. Every new encode records a positive
uniform block size there, so the fallback mounted the same shards with
legacy offset math and could return wrong data. Absent stays legal
(legacy volumes predate the sidecar), and a zero-byte stub still reads
as absent (Go's MaybeLoadVolumeInfo convention, now mirrored in Rust);
a present-but-unreadable or malformed .vif fails the mount instead.
* ec: bound the reconstruct fan-out of one needle's intervals
A degraded interval fans out a read to every reachable shard location, each
with a buffer the size of the interval. Reading a needle's intervals in
parallel multiplied that by the interval concurrency: a needle spanning 8
blocks could hold 8 x MaxShardCount remote reads and buffers at once, where
the sequential version peaked at MaxShardCount. Give each needle a single
reconstruct budget its intervals share, held for the buffer's lifetime, so
separate reads stay independent but one read cannot multiply its own
fan-out.
* ec: drop the duplicated shard-size formula
calculateExpectedShardSize reimplemented the padding rule that
UniformBlockSize already owns — TestUniformBlockSizeMatchesLegacyShardSize
asserts the two agree for every input — so a change to the rule would have
had to be made in both. Defer to the helper, keeping the historic answer for
an empty .dat.
* ec: resolve the shard block layout from whatever records it
Four places still answered the layout question by inference when a record of
it was available, or accepted an answer that was not one:
- A mount with no .vif defaulted to the legacy layout; the bitrot sidecar
records the same config at encode time, so take it when present, as
weed fix -ecx already does. The vif itself is now parsed once per mount
rather than twice.
- The Rust ecx rebuild derived its row count from the padded shard extent,
which under the legacy layout reads a shard that is an exact large-block
multiple as one row too many. Pass the encode-time .dat size from the .vif
and keep the extent as the fallback.
- weed fix -ecx read the block size outside the EC-config guard (collapsing
the unknown sentinel into a definitive legacy), only wrote the recovered
layout back when the .vif was absent rather than unusable, and broke a
scan tie by candidate order instead of the documented reach.
- The uniform layout tripped writeDatFile's large-block ambiguity guard,
which cannot apply when the large and small blocks are the same size.
* ec: give the index-recovery tests a parseable vif
The fixtures wrote the literal bytes "volinfo" as the source .vif and the
recovery copies it verbatim, so the receiving server then mounted the volume
from a .vif it could not parse. That used to pass by silently defaulting to
the legacy layout; a mount now refuses a vif it cannot read, which is what
the tests were exercising all along without meaning to.
* ec: validate the layout a vif records, not just its syntax
Review follow-ups on the mount-strictness change:
- A .vif can parse and still record a block size no encoder could have
produced (negative, or not a whole number of small blocks). Both servers
took it and mapped every read through it. ValidateBlockSize / the Rust
mirror now refuse the mount, the same way an unparseable vif does; 0 stays
valid as the legacy two-tier layout.
- The bitrot-sidecar fallback accepted parity_shards == 0 and summed the
counts in their own width, so values near the ceiling wrapped past the
MaxShardCount bound. Require both counts and sum in a wider type.
- weed fix -ecx treated a config with only DataShards > 0 as usable, so a
half-written .vif suppressed the recovery paths AND survived the rewrite.
Require a complete, in-range config before trusting it.
- Returning the vif-load error left the .ecx and .ecj descriptors open;
repeated mount attempts on malformed metadata could exhaust them.
* ec: refuse to act on a layout the metadata does not establish
- The worker encode only logged a failed .vif write and skipped it in the
distribution set, and treated the .ecsum write as best-effort. A worker
whose disk filled after the much larger shards landed could still
distribute, mount, verify shard inventory, and delete the source replicas —
leaving holders with shards whose geometry nothing records. Both writes and
both inclusions are encode success conditions now.
- A generation-matching .ecsum that disagreed with the .vif geometry only
disabled checksums in Go, and in Rust was not compared at all, so
protection stayed On while reads used the other layout. Both files record
the layout their generation was encoded with, so a disagreement now fails
the mount.
* ec: reject an invalid recorded block size in weed fix -ecx
A .vif with valid shard counts but a negative or unaligned block size was
marked usable: a positive invalid value pinned the scan to a geometry that
de-stripes to garbage, and a negative one ran the dual scan but left the
invalid .vif in place afterwards. Validate it with the same rule the mount
applies, and when it fails leave the layout unknown so the scan recovers it
and the file is rewritten.
* ec: validate the sidecar layout weed fix -ecx recovers from
The .ecsum fallback was taken on DataShards > 0 alone, so a CRC-valid
sidecar carrying the wrong generation, an incomplete ratio, or an unaligned
block size would pin the reconstruction to one incorrect uniform-layout
candidate instead of letting the dual scan decide. Require generation 0, a
complete in-range ratio, and a valid block size; anything less leaves the
layout unknown, which is the answer that still recovers by scanning.
* ec: let only a genuinely absent sidecar choose the legacy layout
With no .vif the bitrot sidecar is the only record of a volume's layout, and
the mount fallback read a failed load, an unusable config, or a sidecar
stamped for another generation as "assume legacy". A uniform generation-0
volume could therefore mount with legacy or another generation's geometry and
answer reads with the wrong bytes. Present-but-unusable now fails the mount;
only actual absence keeps the legacy defaults. Shared as
EcShardConfigFromSidecar so every caller reads the sidecar the same way.
* ec: treat a recorded-but-impossible layout as corruption, not as legacy
- A .vif whose ecShardConfig is PRESENT but records an impossible ratio was
answered with the default 10+4 and the legacy block layout, in both
languages. That reads a uniform volume's shards at the wrong offsets and
returns the wrong bytes. Only an entirely absent config still means "this
predates the record"; a present one that cannot be true fails the mount.
- The shard-count bound summed two uint32 counts as int, which wraps on a
32-bit build: 0x7fffffff + 0x7fffffff lands at -2 and slips under
MaxShardCount. ValidEcShardCounts sums in uint64, and every EC call site
that checked a recorded ratio now goes through it.
* ec: rebuild on the geometry the sidecar records, and flag it when it disagrees
The rebuild RPC passes BackgroundECContext, so RebuildEcFiles resolves the
layout itself — and it resolved a missing or invalid .vif to the default 10+4
with the legacy block size. Two consequences: a 12+4 volume was reconstructed
through a 10+4 matrix, which produces wrong bytes and never regenerates
shards 14-15; and the chosen geometry then contradicted a valid uniform
sidecar, which loadRebuildSidecar reported as BitrotOff — silently skipping
the input and regenerated-shard checksum checks precisely when the volume had
already lost its metadata.
The layout now resolves from the bitrot sidecar (found across the server's
disks, not just beside the base name) before falling back to the defaults,
and a present-but-impossible ratio fails instead of being replaced. A sidecar
that contradicts the chosen geometry is BitrotInvalid, which the existing
unsafeIgnoreSidecar override still lets an operator push past.
* ec: let the Rust rebuild read metadata off a sibling disk
read_ec_shard_config searches only the location the rebuild writes into, so a
volume whose .vif or generation-0 .ecsum sits on another of the server's
disks resolved to the default 10+4 with the legacy block layout — the Rust
half of the geometry-guessing the Go rebuild just stopped doing. It then
reconstructs a custom-ratio or uniform volume through the wrong
Reed-Solomon matrix and de-striping geometry.
The rebuild now looks for the .vif in its own location and then each sibling,
falls back to the generation-0 sidecar wherever that lives, and only defaults
when neither exists anywhere. The encode-time .dat size the ecx rebuild needs
is resolved the same way.
* ec: resolve a rebuild's vif from every directory that may hold it
RebuildEcFiles probed only <data-base>.vif. The caller knows the selected
location's index directory and the sibling locations, but passed neither for
metadata: additionalDirs carried shard directories only, and were searched
for shards and the checksum sidecar. A split -dir/-dir.idx layout, or a disk
holding only shards, therefore resolved a pre-sidecar custom-ratio volume to
10+4 and reconstructed through the wrong matrix — never regenerating shards
14-15.
The caller now hands over the index and sibling directories, and the resolver
probes the vif across all of them, matching what the Rust resolver already
does for both the vif and the sidecar.
* ec: make every rebuild consumer agree on the layout it resolved
- The post-rebuild bitrot backfill re-derived the geometry from this
directory's .vif alone and dropped the block size entirely, so a rebuild
that resolved its layout from a sibling, the sidecar, or a uniform vif wrote
a manifest describing a DIFFERENT layout — one later mounts reject, or that
covers only the default shard count. The layout is resolved once now,
through an exported ResolveRebuildECContext, and the rebuild and the
backfill share that answer.
- The Rust rebuild collected only each location's data directory, so a
sibling's INDEX directory — where a split -dir/-dir.idx layout keeps
.ecx/.ecj/.vif — was never probed, and a custom-ratio volume still resolved
to 10+4 with the legacy layout. Both directories of every location are
carried now, deduped against the rebuild's own.
- A shard delivery can bring the checksum manifest with it, but the receive
path only writes the file: a server that already had the volume mounted kept
its resolved protection state (off) until a remount. The mount RPC
re-resolves it once the shards it describes have been added.
* ec: cover the rebuild's directory search with tests
Reviewers flagged the sibling index directory twice, and the fix that
closed it had no test of its own: the assembly sat inline in the rebuild
handler, reachable only through a gRPC call against a populated store.
Lifting it into rebuildSearchDirs / select_rebuild_location makes the
rule assertable — a sibling contributes BOTH its data and its index
directory, a shared index directory is listed once, and the rebuild's own
data directory never repeats.
Writing the Rust cases surfaced that the two implementations do not agree
on where the rebuild's own index directory belongs, and both are right:
Go's resolver takes a single directory list, so that directory has to be
inside it, while Rust's takes the rebuild's data and index directories as
their own arguments and would search them twice. The tests now state
which contract each side is holding to, so neither drifts into the
other's shape.
Pure refactor otherwise; no behaviour change.
* ec: search the index directory for the layout sidecar
The Rust resolver looked for the generation-0 .ecsum in the rebuild's
data directory and the sibling list, but not in the rebuild's own index
directory — while the .vif lookup directly above it did, and Go's
findBitrotSidecar has always checked both bases. On a split -dir/-dir.idx
location that directory is where the metadata lives, and callers leave it
out of the sibling list precisely because it is passed here separately,
so nothing searched it.
With no .vif anywhere the sidecar is the only surviving record of the
layout. Missing it resolved a 12+4 uniform volume to 10+4 with the legacy
striping — the test added here fails with (10, 4, 0) against the old
code — and the rebuild then reconstructs through the wrong matrix and
writes .ecx offsets that no reader can follow.
* ec: let the rebuild see its own index directory
The Rust rebuild takes a single flat directory list — the shape Go's
RebuildEcFiles uses — so it cannot be handed the rebuild location's index
directory separately the way the layout resolvers are, and the handler
was passing the sibling list, which deliberately omits exactly that
directory. On a split -dir/-dir.idx location that is where .ecx and .vif
live, so the shard and index lookups could not see them.
Go has always carried that directory in additionalDirs; this lines the
two call sites up.
* ec: let a config-free vif fall through to the layout sidecar
A .vif that carries no ecShardConfig answers nothing about the layout, so
it is no more informative than an absent one — but both trees treated its
mere existence as the end of the search. Go went straight to the 10+4
legacy defaults without consulting the sidecar at all; Rust returned
whatever ec_shard_config_from could make of a single directory. A 12+4
uniform volume with a legacy config-free vif therefore resolved as 10+4
legacy, and every read landed at the wrong shard offset.
The sidecar lookup was also single-directory on both sides, while a split
-dir/-dir.idx layout keeps .vif and .ecsum with the INDEX. Go's
findBitrotSidecar has always taken both bases; the callers here passed
only the data base, and the Rust bitrot resolver derived its path from
the data base alone. Rust's layout resolver now takes a candidate
directory list — data, index, then any siblings — and searches all of it,
which also removes the early return that made the vif's presence
decisive.
load_vif_info_across_dirs reported `dir` even when load_vif_info had
found the vif in `dir_idx`. Nothing reads that field today, so this
changes no behaviour; it stops the next caller that resolves the rest of
the volume's metadata against the answer from being sent to a disk
holding none of it.
Absence stays legal throughout: a volume with neither record is genuinely
legacy. Present-but-unusable still fails the mount, now in the
config-free-vif branch too.
* ec: activate a delivered sidecar on every per-disk runtime
A vid mounts as one EcVolume per disk, each with its own resolved
protection state, but the post-delivery reload used the first-match
lookup and so touched exactly one of them. The siblings kept reporting no
protection until a remount — and since shard distribution deduplicates
the metadata files onto the first target disk for a node, the runtime
that got the .ecsum is not necessarily the one the lookup returns.
Iterate every runtime instead, via a new FindAllEcVolumes and its Rust
mut equivalent. Combined with each runtime now resolving its sidecar
against its index directory as well as its data directory, a server
sharing one -dir.idx across its disks activates all of them from the
single delivered copy.
The Rust volume server had no post-mount reload at all; it gets one here,
matching Go.
* ec: resolve the delivered sidecar across every EC metadata directory
Reloading every per-disk runtime, added last round, did not by itself
make the delivered manifest reachable. Startup mirroring copies
.ecx/.ecj/.vif to every shard-bearing disk so each mounts
self-contained, but deliberately not .ecsum, and a repair delivers
exactly one copy. Each runtime was resolving against its own two
directories, so every sibling of the disk that received the file kept
reporting no protection however often it reloaded.
Resolve one authoritative copy across every EC metadata directory
instead of duplicating the file. Mirroring .ecsum would have to keep
pace with a file that is rewritten as shards are repaired, and would not
help the reported case at all: the delivery happens at runtime, and
mirroring only runs at startup.
The regression test pins both halves — a reload restricted to the
volume's own directories still finds nothing, and the same reload
given the server's metadata directories turns protection on.
* ec: ask every directory before writing a TOFU baseline
After a rebuild the opportunistic backfill asks whether this volume
already has a checksum manifest, and answered from the data base alone.
A split -dir/-dir.idx layout keeps the sidecar with the index, and a
multi-disk server may keep it on a sibling, so an existing manifest read
as absent.
The consequence is worse than a missed read. On a false "no" the backfill
writes a fresh sidecar at the data base from whatever the shards say right
now — and the data base is the first candidate every resolver checks, so
that TOFU baseline shadows the real manifest rather than sitting beside
it. A shard that was silently corrupt gets blessed, and the record that
would have caught it stops being consulted.
FindBitrotSidecar exports the search the package already used internally,
so the question is asked of the data base, the index base and the sibling
disks — the same candidates the rebuild resolves its layout from.
* ec: refuse a shard block size no encoder could have produced
weed fix -ecx derived one from the raw shard extent, so a truncated or
partially copied shard wrote a .vif that NewEcVolume then permanently
refuses — the volume the tool was run to rescue could never mount again.
An extent that is not a whole number of small blocks cannot have come
from a uniform encode, so it is no longer offered as a candidate, and
nothing unvalidated reaches the .vif.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: derive the .vif's dat size and block size from one measurement
VolumeEcShardsGenerate stat'ed the .dat before the encode while
WriteEcFiles stat'ed it again to size the blocks. A write landing
between the two produced a .vif whose own two fields describe different
files. WriteEcFiles now leaves both on the context, and fills a
placeholder context in place so the caller can read them back.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: keep the source volume until every holder serves its shard layout
The uniform layout rides in a .vif field older volume servers never
knew: they discard it, mount the shards as legacy and return wrong bytes
with nothing erroring, and the shard files are the same length either
way so no other check notices. The upgrade order lived only in the
release note. VolumeEcShardsInfo now reports the block size the holder
actually serves, in both the Go and Rust servers, and the pre-delete
verification refuses to drop the source unless every reachable holder
echoes the one the shards were encoded with — while a rollback still
exists. A server that predates the field answers 0, which is the
negative answer.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: drop the rebuild's dead block-size parameters
generateMissingEcFiles never reads largeBlockSize/smallBlockSize —
Reed-Solomon reconstruction is layout-agnostic — so passing the legacy
constants only advertised a layout the rebuild does not use. Also move
UniformBlockSize's doc off ValidateBlockSize.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: warn about EC defaults only when the mount used them
The "vif file not found, using defaults" warning fired even after the
bitrot sidecar supplied a non-default layout, sending anyone triaging
wrong bytes after the legacy layout the volume never mounted on.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: stat the distributed bitrot sidecar once
The strict check re-stat'ed the file immediately before the stat that
already gates inclusion, and a failed sidecar write now fails the encode
outright, so the first could only fire on a deletion between the two
lines.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: say what the reconstruct budget actually bounds
A shard's buffer stays in bufs until its interval reconstructs, which is
after the read that filled it released its permit, so the semaphore
bounds round trips in flight and not retained bytes. Peak memory is the
intervals reconstructing at once times the shards each reaches times the
interval size.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* test: let the fake volume server report its delivered EC layout
The pre-delete verification now asks each holder which shard block
layout it serves, and a fake that always answered "unset" looked exactly
like a volume server too old to know the field. Distribution ships the
.vif to every holder alongside its shards, so read the layout back out
of it as a real holder does.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* filter by volume ids
* master: carry the volume ids VolumeList asks about in one repeated field
One id and a list of them ask the same question, so field 2 holds the list
rather than standing beside a second field that supersedes it.
Claude-Session: https://claude.ai/code/session_011qAmAdhrYvnzGkw7A9N4mP
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* Introduce a new `READS` scrub mode.
`READS` performs a full volume scrub but, unlike `FULL`, it will attempt to
reconstruct data for missing/damaged shard intervals from other shards in the cluster
when necessary.
The goal of this check is to ensure that EC volume contents _are readable by Seaweed_
even on a degraded storage state, by exercising parity data which is not read in `FULL`
mode. This is useful not only to validate data is user-readable, but also to detect potential
parity shard issues which may be difficult to pinpoint otherwise - particularly for older
volumes lacking sidecar data, and hence unaffected by `CHECKSUM` scrubs.
For regular volumes, this operation is equivalent to `FULL`.
Example:
```
> ec.shard.unmount --volumeId=1 --shardId=0,3,11 --delete --apply
Live shard topology for volume ID 1 (14 shards):
0@10.200.18.89:9001
1@10.200.18.89:9002
2@10.200.18.89:9003
3@10.200.18.89:9004
4@10.200.18.89:9005
5@10.200.18.89:9006
6@10.200.18.89:9007
7@10.200.18.89:9008
8@10.200.18.89:9009
9@10.200.18.89:9013
10@10.200.18.89:9010
11@10.200.18.89:9011
12@10.200.18.89:9012
13@10.200.18.89:9020
Will unmount + delete 3 shard(s):
0@10.200.18.89:9001
3@10.200.18.89:9004
11@10.200.18.89:9011
Unmounting shard 0@10.200.18.89:9001 for volume ID 1...
Deleting shard 0@10.200.18.89:9001 for volume ID 1...
Unmounting shard 3@10.200.18.89:9004 for volume ID 1...
Deleting shard 3@10.200.18.89:9004 for volume ID 1...
Unmounting shard 11@10.200.18.89:9011 for volume ID 1...
Deleting shard 11@10.200.18.89:9011 for volume ID 1...
All done!
> ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=full
using FULL mode
Scrubbing 10.200.18.89:9002 (1/1)...
Scrubbed 6 EC files and 1 volumes on 1 nodes
Got scrub failures on 1 EC volumes and 1 EC shards :(
Affected volumes: 10.200.18.89:9002:1
Affected shards: 10.200.18.89:9002:1:0
> ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=reads
using READS mode
Scrubbing 10.200.18.89:9002 (1/1)...
Scrubbed 6 EC files and 1 volumes on 1 nodes
```
* ec: report the shards a READS scrub had to rebuild
A READS scrub that recovers an interval was recording nothing, so a volume
missing three shards came back clean and nobody repaired it. The unreadable
shard is now recorded before the rebuild is attempted: READS reports the same
broken shards as FULL and differs only in whether the needles themselves
failed, which is the signal worth having - shards are gone, data is still
there.
forceDeletedNeedlesCheck now applies to READS as well, in the shell and in the
RPC guard: it runs the same needle walk as FULL.
Regenerated the proto instead of hand-editing it, so the pancis typo (which
protoc-gen-go-grpc emits into eight other files here) and the header whitespace
stay as generated.
Mirrors into the Rust volume server, which also now honors
force_deleted_needles_check rather than hardcoding it off.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
* ec: answer a deleted needle from a READS rebuild as deleted
#11020 gave the Rust recovery a deleted flag alongside its bytes, and it
answers a deleted needle with no bytes at all. The READS scrub appended that
empty answer, which does not compile against the new signature and, once it
did, would leave the needle short and report the size mismatch as damage.
Zero-fill the interval instead, the way the direct read beside it already
does: the assembled needle then reaches read_bytes as the delete-state
mismatch the walk already tolerates. Go takes the same branch off the flag
its recovery returns, rather than discarding it.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
---------
Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch>
* master: bound each volume server DeleteCollection, and finish the fan-out
A collection delete fanned out to every volume server holding it with
context.Background(), so a server that accepted the connection and then
went quiet held the whole delete open with nothing to end it. Each RPC is
bounded now, on the same budget allocateVolumeTimeout gives the other
master-to-volume-server admin RPC. The volume server runs the delete to
completion regardless of the request context, so giving up costs the
confirmation and not the deletion.
The walk itself is the caller's, not a per-server one:
- It outlives the caller. A cancelled request must not abandon a
destructive fan-out part-done, with volumes left behind and no request
still running to come back for them.
- It no longer stops at the first server that refuses, which left the
collection on every server after it in the list. The first failure is
still what is reported, and the collection stays in the topology so a
later delete comes back for the rest.
- It sends one RPC per server rather than one per replica.
ListVolumeServers reports a node once for every replica it holds, while
DeleteCollection removes the whole collection from the server it
reaches, so a collection with thousands of volumes repeated the same
whole-collection delete thousands of times over.
Both passes run too. Returning after a failed normal pass left the
collection's EC shards in place with nothing left to retry them.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* master: delete the EC shards behind /col/delete too
The HTTP handler carried its own copy of the volume-server walk and only
ever ran the normal pass, so a collection deleted through it kept its EC
shards. It shares the gRPC path now, which also gets it the bounded RPCs
and the one-per-server fan-out.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* filer: bound the collection delete a bucket delete leaves behind
Deleting a bucket entry deletes its collection afterwards, deliberately
detached from the request so a client that hangs up cannot strand the
bucket's volumes. Detached meant unbounded, though: with the master down
or mid-election the wait for a leader has nothing to end it, so the
handler parks, and the client retrying behind it parks another.
It keeps outliving the request and now carries a deadline of its own. The
budget bounds the wait, not the work: the master keeps deleting on its own
fan-out once asked, so giving up costs the confirmation.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* s3api: bound the collection RPCs a bucket creation and deletion issue
Neither carried a deadline, so a transient failure anywhere down the chain
held the S3 request open until the client gave up on it. Both budgets are
taken outside the filer failover walk, so one budget covers the whole walk
rather than granting each filer a fresh one.
The walk itself stops when that budget is spent, and stops without blaming
anyone: the caller's own expiry is not evidence against the filer that was
answering, and the next filer has no time left to answer in either.
Recorded as a filer failure, a slow master upstream would flag every filer
in the walk, and the three failures that open the circuit take unrelated
object reads down with them.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* s3api: a failed collection listing no longer fails a bucket creation
PutBucket lists collections to notice a leftover one it is about to reuse.
The result feeds a warning and nothing else -- s3a.exists is what decides
whether the bucket already exists -- yet a transient failure of that
listing returned 500 and refused the creation. It is advisory now, so a
failure is logged and the creation continues, exactly as it does when the
listing returns false.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* s3: reject a part whose upload was aborted while its body was in flight
The upload-exists check runs before the part body is read. An abort answered
during the read deletes the upload directory, and the part write that follows
re-creates it, so the aborted upload is listed nowhere yet completes.
Re-check after the write: only createMultipartUpload stamps the destination
key on .uploads/<id>, so a directory without it is one the part write
resurrected. Drop it along with the part and answer NoSuchUpload.
Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT
* s3: reject a copied part whose upload was aborted mid-copy
UploadPartCopy has the same window as UploadPart: the upload-exists check
runs before the bytes are copied, and the part write that follows re-creates
the directory an abort removed. Both the re-encryption and the raw-copy path
re-check before answering.
Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT
* s3: do not complete an upload whose directory holds no upload record
A .uploads/<id> directory that a part write created rather than
createMultipartUpload carries no destination key, no owner and no
encryption settings. Completing one turned stray parts into an object;
answer NoSuchUpload instead.
Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT
* s3: log the part left behind when the resurrected directory survives
abortMultipartUpload can fail to remove what the part write re-created. The
client still hears NoSuchUpload, since the upload is gone either way and a
retry would only write another part, but the leftover is worth a line.
Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT
* ec: split the shard-interval recovery into a gather and a rebuild
Recovering an interval is now one function doing the local seeding, the waved
peer fetch, the shard accounting and the Reed-Solomon rebuild, under a memory
budget. Splitting the gather from the rebuild makes the rebuild a plain
function over a set of intervals, which is testable on its own and reusable by
the parity checks a full scrub wants.
The rebuild refuses a parity target, and the caller checks that before the
gather so a doomed target costs no fan-out. ReconstructData rebuilds data
shards only, so asking it for a parity shard returned no error and left the
slot nil, and the caller copied that out as a successful read of zeroes. Only
data shard ids reach here today, so this is a guard, not a live fix.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
* ec: rebuild only the EC shard the read asked for
ReconstructData rebuilds every missing data shard. The gather stops as soon as
DataShards intervals are in hand, so on a distributed volume it routinely
finishes holding parity where data is missing -- and each of those data shards
is then rebuilt into an interval-sized buffer, decoded, and never read. Ask for
the one shard the read needs.
The budget covers it now too: DataShards gathered plus the one the rebuild
allocates. It never covered the rebuild's output, and with ReconstructData that
output was up to ParityShards buffers.
The required mask is Total() long rather than DataShards. reedsolomon documents
both lengths, but its presence scan walks every shard and indexes the short
mask past its end, so the documented short form panics whenever a parity shard
is absent - which here it usually is.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
* Re-check an EC shard map a failed read has disproved
A read that fails against a cached location drops that shard from the map,
which leaves it one short of complete -- and a map one short is trusted for
seven more minutes. So a moment's trouble between volume servers cost
minutes in which every read of that shard skipped the direct fetch and paid
for a Reed-Solomon recovery instead, at DataShards times the memory and the
peer load.
Mark the map when a read disproves it, and re-check a marked map on the
same eleven-second footing as one that never had enough shards to begin
with. The mark clears on refresh, so it buys one prompt re-check rather
than a master lookup per read. The tiers move into a helper; they were
three overlapping conditions in one expression, and the reading of them
was not obvious.
Rust keeps the entry rather than dropping it -- a dead peer fails fast on
the next attempt, and it was the freshness window, not the entry, hiding a
shard that had moved.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Invalidate the location of an EC shard whose own read failed
Recovery fans out to the other shards, so the one whose direct read just
failed is the only location nothing ever invalidates: a shard that moved to
another server was reconstructed on every read until the map's own window
expired, up to thirty-seven minutes for a map still complete. Mark the map
there too. The entry stays -- a moved shard's old holder fails fast, and
the next refresh is seconds away.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Consume the stale mark before the lookup, not after
A read that fails while the master is answering has disproved the very map
that answer is about to install, and clearing the mark on the refresh's
return swallowed it. Clear it where it is acted on instead. A lookup that
then fails loses the mark, which costs nothing: the refresh time is only
advanced on success, so the next read looks up regardless.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Judge the shard map and consume its mark in one critical section
Reading the mark and clearing it were two separate acquisitions, so a mark
raised between them was cleared by a refresh that had not seen it. In Go
that gap was a few instructions; in Rust the mark was read when the read
first snapshotted the volume and cleared at the decision point, with the
local interval reads in between. Take both under one hold. Rust needs a
mutex rather than an atomic to do it, and no longer carries the mark
through the snapshot.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Put the stale mark back when the lookup does not answer for it
Consuming the mark up front assumed the lookup would supersede it. A
lookup that fails, or comes back with fewer than DataShards holders,
supersedes nothing: the map is unchanged, its refresh time unadvanced, and
with the mark gone the map a read had disproved is trusted for its full
window again on the strength of a lookup that never landed. Put the mark
back on both branches.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* mount: implement fallocate instead of reporting it unsupported
Fallocate answered ENOSYS, so the kernel marked the mount as having no
fallocate and returned EOPNOTSUPP. glibc then fell back to its emulation,
which preads a byte from every block already inside the file to see if it
is allocated; on a write-only descriptor that pread is EBADF, and
posix_fallocate returned it.
Volume space is assigned when a write is flushed, so nothing can be
reserved up front: a range inside the file is answered OK untouched, and
one past the end grows the file the way a truncate would. A mode we
cannot honor is refused with ENOTSUP, not ENOSYS, so the kernel keeps
sending the ones we do.
Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9
* mount: let a fallocate that allocates nothing past the quota and worm guards
A range already inside the file, and any FALLOC_FL_KEEP_SIZE request,
reserve no space and rewrite no entry, but the preflight refused them
with ENOSPC on a full mount and EPERM on a worm-enforced file. Decide
the no-op first and guard only the growth.
Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9
* mount: charge a fallocate growth to the uncommitted byte counter
Write charges the counter by how much the file grew, so the writes that
fill a range fallocate already extended charge nothing and the real-time
quota check never sees that data — only the periodic filer refresh does.
Count the growth where it happens.
Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9
* mount: charge a truncate-up growth to the uncommitted byte counter
Same gap Fallocate had: Write charges the counter by how much the file
grew, so the writes that fill a range ftruncate already extended charge
nothing and the real-time quota check never sees that data. Count the
growth where it happens; a shrink still leaves the counter alone, since
it is only ever raised and then reset by the periodic filer refresh.
Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9
* s3: stop retrying a delete the filer refused for a non-empty folder
The filer looked and the children are there, so the answer will not change.
retryFilerOp spent six attempts and up to 3.1s of backoff on it before the
caller could act on the condition it was already holding.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: thread the request context through the unversioned delete path
doDeleteEntry issued every DeleteEntry on context.Background(), so an S3
client that hung up left the gateway working on its behalf, out of reach of
both cancellation and the per-request retry allowance that
DeleteMultipleObjectsHandler installs.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: treat a cancelled filer RPC as terminal, not transient
isRetryableFilerErr matched context.Canceled and DeadlineExceeded by
sentinel, which only holds while the error is still local. Once it has
crossed gRPC it is a status, so an abandoned request was retried six times
on behalf of a caller that had already gone.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: replay a delete whose reply the transport dropped
A delete is idempotent at the filer, which answers an entry that is already
gone with an empty resp.Error, so a reply lost in transit can be reissued
rather than surfaced. Surfaced, it becomes a 500 on the bucket delete, which
boto3 resends and is then answered NoSuchBucket, or a per-key InternalError
inside the 200 of a multi-object delete, which no SDK retries at all.
The replay runs through retryFilerOp, so it draws on the allowance the
request already installs rather than paying a backoff per key, and stops for
a caller that has gone. rm and rmObject re-enter WithFilerClient per attempt,
so each one walks the failover list again on a connection the failed attempt
had invalidated; the multi-object loop holds one client for the batch, so
there the replay reuses it.
Classification stays structural. The filer reports its own refusals in
resp.Error, which carries no status and has the deleted path - and, for a
recursive delete, the children it stopped on - formatted into it, so no key
name can steer the decision either way.
rm and rmObject now take the caller's context. Cleanup and rollback paths
pass context.Background() deliberately: they have to run whether or not the
caller is still waiting.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: share one retry allowance across multipart completion cleanup
The unused-entry loop deletes once per entry, and each delete now retries,
so a filer that stays unavailable held the response for 3.1s per entry after
the object was already committed.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* Replay metadata log chunks the way the mount reads every other chunk
The subscription's log-chunk replay built its own lookup, which always
resolves volume server addresses. A mount started with
-volumeServerAccess=filerProxy cannot reach those, so every fresh
subscription failed on the previous minute's persisted segment and
resubscribed a second later, forever. Take the lookup from the caller
instead; the mount hands over the one it uses for file reads, which also
keeps publicUrl and the bounded location cache in play.
Claude-Session: https://claude.ai/code/session_01NGqrxYj7cHUpSrL249n3Z6
* Keep a log chunk read failure off the filer connection
A metadata subscriber reads persisted log chunks over HTTP from volume
servers and hands whatever went wrong back as the subscription's error.
"connection refused" from a volume server then matched the transport
patterns that decide a gRPC channel is dead, so every failed replay
closed the shared filer ClientConn and cancelled the assign and upload
RPCs riding on it with "the client connection is closing". Mark those
read failures so they are judged for what they are.
Claude-Session: https://claude.ai/code/session_01NGqrxYj7cHUpSrL249n3Z6
* Reconstruct an EC shard from the shards already on this server
recoverOneRemoteEcShardInterval only ever fanned out to the cached shard
locations, so a server holding shards of the volume still fetched them
over gRPC from itself -- and when the peers were unreachable it could not
reconstruct at all, even holding the whole volume on local disk. Seed the
Reed-Solomon buffers from the locally mounted shards first; each one is a
peer round trip, and an interval-sized buffer, the fan-out no longer needs.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Fetch only the EC shards reconstruction still needs
The recovery fan-out read every surviving shard, so a 10+4 volume pulled
13 interval-sized buffers to feed Reed-Solomon 10 -- a third more memory
held, and a third more load asked of peers that were, by definition,
already having trouble. Fetch what is missing, and widen only when some of
those reads fail. A shard reporting the needle deleted ends the walk: the
rest would only answer the same.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Bound the bytes EC recovery holds in flight
Recovery is the one read path that multiplies the served bytes: it holds
an interval-sized buffer per shard until Reed-Solomon runs, and a peer that
is slow to fail keeps them all alive for the whole gRPC timeout. Nothing
bounded how many of those fan-outs ran at once, so a transient problem
between volume servers turned every read into a DataShards-fold allocation
and the server died of it -- 64 concurrent 4MB intervals pin 3.6GB, and
that is a small burst.
Charge each recovery against a process-wide budget, so a burst queues on
the semaphore instead of on the heap.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Answer a deleted EC needle as deleted, not as a failed recovery
A holder reporting the needle deleted is authoritative: deletes are never
invented and never undone. Recovery already collected that flag, then
dropped it on the branch where too few shards came back -- so a read of a
deleted needle that had to recover surfaced as "cannot recover shard", and
the volume server answered 500 where it owed a 404. Carry the flag out of
the shortfall, and let it decide ahead of the error it came with.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Check the encode run of a locally seeded EC shard in Rust
The Rust recovery seeded Reed-Solomon straight from the mounted shards,
without the encode-run check the remote reads and Go's
readLocalEcShardInterval both apply. A volume remounted from a newer
encode between the read's snapshot and its recovery would have fed
mixed-generation bytes into the reconstruction.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Say what the recovery budget actually guarantees
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Seed Rust EC recovery from shards on every local disk
find_ec_volume returns the first disk's EcVolume, so a reconciled volume
whose shards are split across data dirs had the siblings ignored and could
report "cannot recover" while holding enough shards locally. Resolve each
shard together with the disk that owns it, the way Go's recovery already
does, and check that owner's encode run.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Name the failure when the source cluster cannot locate a chunk's volume
LookupFileId formatted a nil err into the message it returned, so the only
thing a caller could do with "no locations for this volume" was match on the
text. Return a typed error instead.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Fail a source chunk read on a failure status instead of copying the error page
ReadPart never looked at the response status, so a volume server answering 404
for a needle vacuum had removed came back as a successful read whose body was
the error page. The caller counted those bytes as file content and reported a
size mismatch — a corruption claim about data the source had simply lost — and
a 404 from one replica ended the search instead of trying the next.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Stop retrying a chunk the source cluster can no longer produce
A chunk whose volume vacuum has removed fails the same way on every attempt, but
the retry loop had no way to say so and kept going forever. The sync job holding
it never finished, so it pinned the offset watermark at the event ahead of it and
filer.sync never checkpointed again — alive, quiet, and permanently behind.
Wait the source out for a grace period long enough to cover a volume server
restart or a master failover, then give up and mark the failure permanent.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Let replication continue past an entry whose source data is gone
An entry the source can no longer read holds the sync offset forever: the event
fails on every replay, so the checkpoint never moves past it and every later
event stays uncheckpointed, however long the sync keeps running. Nothing brings
those bytes back, so skip the entry with an error naming it and carry on.
Skip only while the source is demonstrably still serving other chunks. A volume
with no locations reads the same whether it was vacuumed away or every replica is
down, and during a cluster-wide outage that answer comes back for every chunk —
skipping then would drop live files wholesale.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Propagate a missing source chunk instead of waiting when supersession is unverifiable
An incremental sink's dated target keys cannot be mapped back to a source path,
so nothing here can tell a chunk the source lost from one a later version already
replaced. Waiting out the grace period would stall every vacuumed needle for half
an hour; hand the failure to the caller, which has the event's real source key.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Wait out a gone volume once, not once per file it held
A volume vacuum removed took every file it held with it, and each chunk was
timing its own grace period. With a bounded chunk executor those waits serialize,
so one gone volume holding many files stalls the sync for far longer than the
grace period — the wedge again, only slower.
Track the wait per source volume on the sink instead: the first chunk to find it
unlocatable starts the clock, every later chunk inherits it and gives up as soon
as it has run out, and a chunk the source does serve clears it.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Probe the source with a read, not a lookup, before writing an entry off
A lookup only proves the source master still has the topology. If every volume
server is unreachable while the master still lists them, the probe passed and the
sink wrote off an entry whose data was merely out of reach. Read the probe chunk
instead, and say in the log that the entry stays unreplicated.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* util, pb: classify a filer error by the status the server sent
DoSeaweedListWithSnapshot wrapped a failed ListEntries with %v, dropping the
gRPC status, so IsTransientError fell back to matching substrings against a
message that now held the caller's path. Keep the status with %w and let it
decide, reading the server's own text rather than the wrapper's.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* s3: keep the bucket and prefix out of the list retry decision
A bucket named transport, or a prefix under logs/unavailable/, made a
PermissionDenied listing look transient and got it retried; a key holding the
not-found sentence suppressed a retry that should have run. Both checks now
read the filer's status, and only fall back to the text when there is none.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* filer, s3: classify a delete failure before the path is wrapped into it
The filer put the non-empty-folder marker behind its own "delete directory %s"
wrapper and the gateway matched it as a substring, so a key named after the
marker turned a real delete failure into the demote-the-marker no-op and the
request answered 204. Keep the marker leading the message that crosses the
wire, turn it back into a sentinel where the response is read, and match that.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* s3tables: report a delete the filer rejected
deleteDirectory discarded DeleteEntryResponse and checked only the
transport error, so DeleteTable, DeleteNamespace, DeleteView and
DeleteTableBucket answered 200 for a delete the filer refused. Call
filer_pb.DoRemove, which reads resp.Error and still treats a missing
entry as success.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* admin: report a delete the filer rejected
The bucket delete, the file browser handlers and the topic retention
purger all discarded DeleteEntryResponse, so a delete the filer refused
came back as success. Call filer_pb.DoRemove, which reads resp.Error.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* credential: report a delete the filer rejected
DeleteUser, DeletePolicy and the full-sync cleanup loops discarded
DeleteEntryResponse, so a rejected delete answered success and left the
credential file in place. The service account path in the same store
already read resp.Error; the rest now do too, via filer_pb.DoRemove
where not-found is already tolerated.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* shell: report a delete the filer rejected
remote.configure -delete, remote.cache and the remote metadata sync
discarded DeleteEntryResponse, so a rejected delete printed as removed.
Call filer_pb.DoRemove, which reads resp.Error.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* mq: report a delete the filer rejected
The consumer offset group purge and the coordinator assignment delete
discarded DeleteEntryResponse. Call filer_pb.DoRemove, which reads
resp.Error.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* iam: count only the revocation entries the filer actually deleted
The expiry sweep discarded DeleteEntryResponse, so a rejected delete was
counted as purged and the entry stayed. Call filer_pb.DoRemove, which
reads resp.Error, matching the role and provider stores beside it.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* mount: fail rmdir when the unary fallback delete was rejected
The streaming branch turns DeleteEntryResponse.Error into an error, the
unary fallback dropped it, so rmdir of a non-empty directory answered OK
off the stream and ENOTEMPTY on it. Surface it in both.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* s3tables: fail DeleteTableBucket when the directory delete is refused
The handler only failed when both the leaf entry and the directory
delete failed, so a refused bucket directory delete still answered 200
with the bucket in place. The directory is the bucket, so it decides;
the leaf entry stays best-effort.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
Every key in a multi-object delete drives its own retryFilerOp, so a filer
that is briefly unhealthy multiplied one op's ~3.1s of backoff by a key
count the client picks. The batch now carries a single allowance in its
context, sized to one op's worst case; once it is spent the remaining keys
fail fast with a per-key error instead of holding the request goroutine.
A single-object delete carries no allowance and keeps its full retries.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: bound the wait for a master leader by the caller's context
WithClient waited on GetMaster with context.Background(), so a caller that
arrived while no master leader was known parked in a 200ms poll loop until one
appeared, whatever deadline it had already set on the RPC. Each retry above it
then left another goroutine in the same wait.
Take the context in WithClient and WithClientCustomGetMaster and hand it to
GetMaster, and stop the retry loop once it is done. The dial keeps
context.Background(): fn brings its own RPC context, so a cancellation seen
here cannot be attributed to the shared connection.
Call sites pass whatever they hold: the request context in the filer's
CollectionList, DeleteCollection and Statistics handlers and in the credential
store's propagation, the operation context in the shell's s3.bucket.delete and
the kafka gateway's broker and filer discovery, and context.Background() where
there is none - the shell commands, the admin dashboard wrapper, and the
exclusive locker's initial lease. The locker's release keeps its own
uncancelled context so a slow unlock cannot turn into a ghost lock.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that WithClient gives up with the caller's context
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: cut the master retry backoff short when the caller gives up
util.Retry sleeps unconditionally between attempts, so a transient error
arriving just before the caller's deadline still cost it a full backoff step.
Use the context-aware util.RetryWithBackoff, the same helper the volume lookup
in this file already uses.
Two call sites went with it: the shell's lock-holder lookup builds its three
second bound before WithClient so it also covers finding the leader, as its
comment already promised, and the filer's post-delete collection cleanup goes
back to an uncancelled context - the entry is already gone, so a caller that
hung up must not leave the collection behind.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that a cancel during backoff ends the retry
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
HeadBucket treated any lookup error as ErrNoSuchBucket, so a transient
filer failure answered 404 instead of 500 and clients stopped retrying.
Split the two cases the way the bucket policy handlers already do.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* volume: stop the .vif guard depending on which entry the scan handed over
A volume has both an .idx and a .vif, and loadExistingVolume skipped a .vif
next to an .ecx as EC shard metadata. That was only ever correct because
os.ReadDir sorted .idx ahead of .vif: an interrupted encode, where the .idx is
still there, has to reach validateEcVolume to be reclaimed. Ask for the .idx
instead of trusting the order.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: walk volume directories in batches instead of listing them whole
os.ReadDir builds, and sorts, a slice of every entry before the caller sees
the first one. A disk holding millions of volumes has a .dat, .idx and .vif
per volume, so each startup scan costs hundreds of MB of peak heap that the
runtime is slow to hand back -- and there are several of them before the
first volume loads.
Walk in batches instead, and keep only the entries each scan acts on:
loadAllEcShards now sorts and stats the shard and index files alone rather
than every file on the disk.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: skip the sibling-.dat scan when no EC volume is loaded
pruneIncompleteEcWithSiblingDat only ever prunes EC volumes that are loaded,
but it first walks every disk and keys a map by every .dat on the server. On
a store with no EC volumes at all that is millions of map entries built to
answer no question.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: stop keeping a departure message for every volume
The report state held a VolumeShortInformationMessage per volume copy so a
departure could be named, but almost no volume ever departs. Hold a handle to
the identity instead -- volumes share very few distinct ones -- and build the
message on the way out.
Measured over a populated report state: 195 -> 83 bytes per volume.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* rust volume: stop keeping a whole volume message per volume held
The send loop kept a VolumeInformationMessage for every volume just to notice
mounts and unmounts, and rebuilt the map from scratch on every beat. Keep the
identity a delta names, which is what the Go report state keeps for the same
reason.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* rust volume: keep only the EC files the shard scan acts on
load_all_ec_shards named every file on the disk twice -- once in the dedup set
and once in the sorted vector -- before deciding it only wanted .ec?? and .ecx.
Filter while reading instead. Mirrors the same change in loadAllEcShards.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: share the strings every .vif repeats
A tiered volume's .vif names its replication and its backend, and every decode
allocates a fresh copy, so a server holding millions of them holds millions of
copies of the same handful of names. Route them through the interning table
the volume info decode already uses. The remote key names one volume and is
left alone.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
The router matches bucket subresource routes in registration order while
the IAM action resolver matches its own list in a different order, so a
request carrying two operation subresources is authorized as one
operation and served as another. `PUT /bucket?policy&tagging` resolves to
s3:PutBucketTagging and runs PutBucketPolicy, letting an identity
delegated bucket tagging install an arbitrary bucket policy. The same
mismatch reaches PutBucketCors, PutBucketLifecycle, PutBucketVersioning,
PutObjectLockConfiguration, PutBucketRequestPayment and the policy and
cors deletes.
Reject the ambiguity where the other pre-routing checks live, so neither
list has to stay in step with the other. Keys that modify an operation
rather than select one -- versionId, partNumber, prefix -- still combine
freely.
* shell: remove the directories emptied by volume.fsck's filer entry purge
volume.fsck -findMissingChunksInFiler -reallyDeleteFilerEntries deleted the
orphan entries but left their parent directories behind, so a namespace
accumulated empty directories that had to be cleaned up by hand.
Remember the parent of every purged entry and, once the purge is done, walk
up from each one deleting the directories that are now empty. The delete is
non-recursive, so the filer itself rejects a directory that still has
children; a bucket and a directory that is an S3 object of its own are left
alone.
Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc
* shell: keep a directory volume.fsck saw change under it
The empty-directory sweep read the entry to spot an S3 directory key object
and then deleted unconditionally, so a directory promoted to an object in
between was removed anyway.
Delete with the mtime the lookup returned, leaving the filer to skip a
directory that has changed since.
Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc
* shell: leave a directory volume.fsck just saw written for the next run
The mtime the delete is conditioned on has second resolution, so a write
landing in the same second as the one already on the directory is
indistinguishable from it and the directory would still be deleted.
Skip a directory modified within the last few seconds. A write after the
lookup then always carries a later second than the one the delete carries,
and the sweep picks the directory up on the next run.
Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc
* shell: skip a directory volume.fsck cannot condition a delete on
A zero mtime disables the delete's condition at the filer, so a directory
whose entry carries none was removed unconditionally and a concurrent
promotion to an S3 object went with it.
Leave such a directory alone.
Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc
* shell: hold volume.fsck's quiet period to the cutoff second itself
Mtime keeps whole seconds, so a directory whose mtime lands on the cutoff
second was written up to a second after it. Skip that directory too, so the
quiet period fails closed.
Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc
* s3: enforce object lock when deleting a directory marker
The key "dir/" is deleted the unversioned way, ahead of the branches
that enforce Object Lock, so a principal with plain delete permission
could remove a key the gateway was reporting as COMPLIANCE-retained --
retention set through PutObjectRetention is stored on the directory
entry and served back by GetObjectRetention, only the delete ignored it.
The same path also takes any key ending in "/" regardless of size, while
a PUT only makes a marker of one up to 1KiB. A larger one is a genuine
versioned object, and deleting it here dropped its whole history after
the versioned delete of the same key had been refused.
Enforce in the marker delete itself, so the single, versioned and
multi-object delete paths are all covered.
* s3: apply object lock headers on a directory marker PUT
The trailing-slash branch runs before the versioning and Object Lock
handling, so it accepted x-amz-object-lock-* headers and stored none of
them: a bucket owner could believe a key was retained while nothing
recorded it, and an invalid mode or a past retention date that a regular
key rejects came back 200 here.
Validate the headers the way the regular path does, store what they ask
for beside the owner the same callback already sets, and refuse to
replace a key that is already retained.
* s3: check every version a marker delete would remove
The marker delete clears any history under the key in one recursive
removal, while the lock check ahead of it resolves the latest version
only. A version retained under an unretained one was taken with the
rest, so enforce against each version the removal covers.
* test: pin the marker lock refusals to AccessDenied
A bare require.Error passes on any failure, including one that has
nothing to do with the lock. Assert the code, the key the batch delete
reports, and that the marker survives each refusal.
* s3: check the history entries a version list leaves out
The version list skips an entry without a version id, while the removal
takes it with the rest, so an entry an older build left unnamed escaped
the check. Walk the history directly instead, and refuse when an unnamed
entry is still under a retention or a legal hold of its own.
* s3: let a governance bypass reach an unnamed history entry
The unnamed branch refused every active retention, so a caller allowed
to bypass governance could not clear one, which the named path lets
through. Refuse a legal hold and compliance mode as before, and take the
bypass into account for governance.
* s3: keep the object lock decision in one place
The unnamed history entry had to repeat the retention and legal hold
rules inline because the enforcement helper only takes a key to look up.
Split the part that judges an entry out of it and call that from both.
* s3: guard a marker PUT on the entry it replaces
The overwrite check resolved the key's latest version, but mkdir builds
a fresh entry for the marker itself, dropping the lock metadata the old
one carried. Once the key had a history, an unlocked version answered
for a retained marker and a plain PUT replaced it. Judge the entry the
write is about to replace instead; a versioned write of the same key
still adds a version, which is its own to allow.
* s3: guard a marker delete on the entry it removes
The check ran against the key rather than the entry, so once the key had
a history it answered with a version and the retention recorded on the
marker itself went unseen. Judge the entry that is about to be removed,
the same way the PUT side now does; the versions under it are still
covered by the walk that follows.
* s3: take the object write lock for a marker PUT
The overwrite check read the entry that the mkdir after it replaces, so
two marker PUTs could both pass while one was still unlocked. The marker
delete already runs under this lock; hold it across the check and the
mkdir so the entry cannot change in between, and so the two paths are
serialized against each other.
DeleteTable authorizes the named table, then recursively purges the data
path derived from its stored MetadataLocation. That location is supplied
by the caller at create/register time and never bound to the table, so a
tenant allowed to drop one table could point it at a table in a sibling
namespace and have the delete destroy that table's catalog entry and
data files.
A legitimately decoupled location -- a rename source, or a leftover the
name was reused over -- has had its catalog attributes stripped, so a
surviving metadata marker identifies a path that belongs to another
entry. Refuse those, alongside the existing ancestor refusal.
* volume: build the guarded remote storage client through a shared helper
Fold the endpoint validation, credential check and rebinding-safe dialer
that FetchAndWriteNeedle applies before dialing a caller-supplied remote
storage endpoint into a single BuildGuardedRemoteStorageClient helper, so
other callers that dial the same endpoints can reuse it. No behavior
change on this path.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* filer: build the remote-mount stream client through the guarded helper
streamFromRemote serves a cold remote-only entry straight from its mounted
origin. Build its client through BuildGuardedRemoteStorageClient so the
same endpoint checks the volume server applies cover this read path too.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* s3: build the remote-mount stream client through the guarded helper
openRemoteStream serves a remote-mounted object straight from its origin
when the local read cannot. Build its client through the same guarded
helper so the endpoint checks apply here as well.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* s3tables: share one CreateTable authorization gate
CreateTable and RegisterTable each carried their own copy of the name
validation, policy load and permission check. Fold them into
authorizeCreateTable, and expose it on the Manager for callers that write
into a table bucket before the table itself is registered.
Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy
* iceberg: authorize a table create before it writes
Stage-create returns before the S3Tables registration that authorizes a
create, and the plain create writes its metadata file before reaching it,
so a caller who may not create the table could still leave a staged
template, a marker and a v1.metadata.json in the target bucket - and get
vended credentials for a location of their choosing. Run the CreateTable
gate as soon as the table is known to be absent.
Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy
* iceberg: authorize a create-on-commit the same way
A commit against a table that does not exist creates it, writing the
metadata file first and only then reaching the registration that checks
the caller may create it. Denied callers saw a 500 for what is a 403.
Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy
* iceberg: pin that identity actions reach the create gate
The manager request is built from the caller's own context, so an identity
whose actions carry the permission still passes. Worth a test: a fresh
context here would silently deny every such caller.
Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy
* mount: serve metadata ops from the open handle of an unlinked file
ftruncate on a descriptor whose file was unlinked failed with ENOENT:
maybeReadEntry resolved the inode to a path first, and unlink had already
dropped it. GetAttr worked around that with its own handle fallback;
SetAttr and the xattr handlers had none.
Look the handle up first and let it answer whether or not a name still
points at the inode. GetAttr keeps reporting nlink 0 there, now off the
empty path.
Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega
* mount: read an open handle's attributes under the handle lock too
GetAttr held only the LockedEntry lock, which covers the async uploader's
chunk appends but not Write or the metadata flush: those rewrite size,
times and the whole chunk slice under the handle lock, so FileSize could
walk a slice mid-reassignment. The branch this replaced took both locks;
take both here, outer handle lock first, as Read and Lseek do.
Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega
* mount: report nlink 0 from SetAttr for an unlinked open file
The kernel caches the attributes a SETATTR reply carries, so an ftruncate
on an unlinked file left fstat reporting nlink 1 until the cache expired,
even though GetAttr had it right. Both replies go through the same rule.
Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega
* webdav: name the entry, not its path, in a listing
DAV:displayname carried the full path of every entry. A client that
takes displayname for the child's name - Windows Explorer does - then
looks for /dir/name under /dir and finds nothing, so a folder shows up
empty while the root, where the two spellings differ only by a leading
slash, still lists.
Readdir now builds its entries with toFileInfo like stat does, so a
listing and a lookup describe a child the same way, and the wrapper that
was trimming the sub-folder back off a name goes away with it.
Claude-Session: https://claude.ai/code/session_01XCeuCWpF9xo9CfyHvCQE9c
* webdav: derive an ETag when nothing hashed the entry
Uploads through this gateway carry no content MD5, so filer.ETag comes
back empty and every file in a PROPFIND answered with an empty
DAV:getetag, which is not a valid entity-tag. Report it as unimplemented
instead, the way the sub-folder wrapper already did, and webdav falls
back to modification time and size. The wrapper's copy went with it - it
swallowed the stat error a caller was meant to see.
Claude-Session: https://claude.ai/code/session_01XCeuCWpF9xo9CfyHvCQE9c
* s3: keep a missing object a 404 under If-Match and If-Unmodified-Since
GET and HEAD resolved the target before evaluating the conditional headers, and
a missing target failed If-Match and If-Unmodified-Since outright, so absence
surfaced as 412 PreconditionFailed. AWS reports the missing object instead:
404 for HeadObject, NoSuchKey for GetObject, and 412 only when a live object
fails the condition. Clients cannot tell absence from a stale precondition
without an extra racy HEAD, so OpenDAL disabled its four conditional
stat/read capabilities against SeaweedFS.
A precondition now only fails against an object that exists; a missing one --
including a latest version that is a delete marker -- returns NoSuchKey.
Claude-Session: https://claude.ai/code/session_01X4kEbuwxd9DFsTnSXjfjgv
* s3: evaluate a conditional read against the version the request names
GET and HEAD resolved the latest version before evaluating the conditional
headers, so a request carrying versionId had its If-Match compared against a
different version than the one it was asking for: a live version whose ETag the
client held failed once a newer version -- or a delete marker -- became the
latest. resolveObjectEntry now resolves the named version on a versioned bucket,
the way DELETE already does.
A named version that resolves to nothing is left to the handler, which alone
knows whether the bucket is versioned and so whether it owes NoSuchVersion.
Claude-Session: https://claude.ai/code/session_01X4kEbuwxd9DFsTnSXjfjgv
* volume: resolve the disk IO slow-latency threshold per disk
volume.toml keys [volume.disk.io.slow.latency] by disk type, but the
threshold was chosen once per server by switching on the raw -disk flag.
-disk is comma-separated, one entry per -dir, so a multi-disk server
matched no case and silently took the hdd threshold.
Carry the table on DiskIOProbeConfig and resolve it in CheckDiskSpace
from the location's own DiskType. A type with no entry keeps falling
back to the hdd threshold.
* volume: run the disk IO probe on multi-directory volume servers
The probe was disabled whenever more than one -dir was configured,
because a single server-wide slow-latency threshold could not describe
disks of different types. The threshold is per disk now, and the rest of
the probe already is: diskRegistry is keyed by directory, each
DiskLocation runs its own CheckDiskSpace, and Store consults
isDiskUnavailable per location.
* volume: reject duplicate -dir entries
Nothing deduplicated -dir, so the same directory listed twice produced two
DiskLocations that each loaded every volume in it, appending to the same .dat
under two independent locks. Compare directory identity with os.SameFile
rather than the path, so a symlink or bind mount aliasing an earlier entry is
rejected as well.
* volume: cover the per-disk slow-latency handoff
SlowLatencyFor has a test, but nothing asserted that CheckDiskSpace feeds it
the location's own disk type. Probe through a seam so the resolved threshold
is observable, and check hdd, ssd, nvme, the empty type, and an unlisted tag.
* mount: let a rename remove its source at the source's own version
A rename stamps the source and takes the name away at the same log position,
so the removal reaches the meta cache carrying exactly the version the source
already records. The version gate read that as a write already reflected and
dropped it, while the destination half of the same event still applied -- the
source stayed cached beside the destination, and readdir and stat went on
serving a name the filer no longer had:
gate dropped removal of /winfsp-test-TestRenameOverExisting/src
eventTs=1787761708173717200 record=1787761708173717200
floor=1787761708173717200 tombstone=false
A removal asks a different question from a write. An entry still present at
exactly that version has the write reflected but not its removal, so only a
strictly newer record fences one out; a tombstone is the removal already
reflected and goes on fencing as before.
* mount: sweep a section's vanished name recorded at the snapshot
The refresh deletes the names its listing did not return, but asked the gate
whether a write at the snapshot was reflected. A name recorded at exactly that
version has the write reflected and not its removal, so it survived the sweep
and stayed cached until some later event happened to touch it.
Same reading as the rename source a commit earlier: the call site removes, so
it asks about a removal.
* s3: keep verifying the request host when externalUrl is set
externalUrl was the only host candidate once set, so a client that dialed
the gateway directly instead of through the proxy always got
SignatureDoesNotMatch. Make it lead the candidate walk instead: every
candidate still needs a valid signature, and the request-derived hosts are
already trusted when the flag is unset, so a mixed proxy plus in-cluster
topology can now advertise a public endpoint and verify both planes.
* s3: cover virtual-hosted addressing behind externalUrl
The old pin also rejected an external client that signed
bucket.api.example.com, since only the bare externalUrl host was ever
tried. The candidate walk covers it; pin the case down.
* operation: give an encrypted chunk the plaintext ETag
With -encryptVolumeData the volume server stores ciphertext, so it cannot
echo a Content-MD5 back and the chunk lands with an empty ETag. Every ETag
derived from those chunks then comes out empty for a single chunk, or
d41d8cd98f00b204e9800998ecf8427e-N for several.
The caller already hashes the plaintext to send as Content-MD5, so keep that
digest as the chunk ETag instead of dropping it, and compute it for a
WantMd5 caller under cipher too.
* s3: re-encrypt a part copy from a volume-encrypted source
UploadPartCopy raw-copies source chunks when neither side uses SSE, which
also caught -encryptVolumeData sources. Those chunks are ciphertext a
whole-chunk cipher key decrypts, so copying a byte range out of one and
keeping the key leaves a destination that fails authentication on GET, and
the copied chunks carry no ETag for the part result to report.
Route them through the re-encrypting path already used for SSE: it reads the
source as plaintext, hashes the part, and writes the destination under the
gateway's own encryption.
* s3: fetch only the range a part copy asked for
The re-encrypting UploadPartCopy path opened the source at offset 0 and threw
the prefix away, so assembling an object part by part read the source once per
part. Now that volume-encrypted sources take this path too, that is the common
case rather than an SSE corner.
The chunk stream already seeks, so hand it the range.
* s3: reject an unsatisfiable copy-source-range
A part copy has no way to report a short part, so a range reaching past the
source cannot be clamped the way a GET clamps one. The fast path silently
produced a part shorter than asked for, or an empty one; the re-encrypting
path pads with zeros, so a 2 MiB source copied as bytes=1048576-9999999 came
back as 1 MiB of data followed by 7.5 MiB of nothing.
Answer InvalidRange instead, which is what s3-tests'
test_multipart_copy_invalid_range expects.
The test truncates a good .sdx and asserts the result still looks fresher
than its .idx, on the reasoning that truncation bumps the mtime. That holds
only at the filesystem's timestamp granularity: where both writes land in the
same tick the precondition fails and the run reports a failure that says
nothing about the code under test — as it did on CI. Backdate the .idx the
way the sibling stale_sdx_is_regenerated already does.
* mount: stop a replaced rename destination from flushing over the rename
Rename replaces whatever the destination held, which deletes that entry, but
only the source handle was told. A handle still open on the replaced entry
went on flushing its metadata under that name, and on Windows -- where the
close carrying the flush runs after the application's CloseHandle has already
returned -- the flush landed after the rename and put the destination's old
content back:
dir Rename old_entry:{name:"src"} new_entry:{name:"dst" ... inode:...3416}
doFlush /dst fh 1521468582993181449
/dst saveToStorage 1,6872462993 [0,3)
flushMetadataToFiler /dst inode 11939747521756968515
InsertEntry /dst
The next read of the destination returned the content the rename was supposed
to replace. Unlink already handles this with markHandleDeleted, which raises
the flag under the handle's flush lock so a flush already writing finishes
first and any later one sees it; a rename that replaces an entry deletes it
just the same, so it now does likewise.
Verified on the Windows runner: TestRenameOverExisting 300/300, where the same
loop reproduced the corruption twice without this.
* test/winfsp: say which layer kept a renamed-away name
The failure only reported the stat. Which layer answered narrows the search a
lot: a listing reads no per-path cache, the mount's own forgets within a
second, and a name that survives both is still in the meta cache.
* mount: keep the destination barrier honest when the rename does not happen
Two gaps in the barrier the previous commit put in front of a replaced rename
destination:
The flag was raised before the filer rename, which can still fail. The
destination then stays exactly where it was, with its handle marked deleted
and its dirty metadata silently dropped from then on, so a rename that
returned an error has to put the flag back.
The handle was only found through the path mapping, which Forget drops while
the handle is still open. The source side already falls back to the inode the
entry carries; the destination now does the same, off the entry the sticky-bit
check had already loaded.
* mount: let only the caller that raised a delete mark lift it
Restoring the destination handle after a failed rename cleared isDeleted
outright, so an unlink that marked the same handle in between lost its mark and
a later flush could write the unlinked entry back.
Every raise of the flag already happens under the handle's flush lock, so
counting them there is enough to tell one caller's mark from another's: the
rename lifts only the mark it made itself.
* mount: drain the destination flush before marking it deleted
A flush already queued for the destination belongs to the entry as it stands.
Marking first meant the drain waited on a flush that then skipped its metadata
as deleted and released its handle, so a rename that failed afterwards had
nothing left to restore and the queued update was gone, its chunks orphaned.
Draining first lets that flush finish as itself, before the rename has taken
anything away.
* filer: cover the storage rule TTL on the object transaction write path
An object written through ObjectTransaction used to land with ttlSec 0
even under an fs.configure TTL rule, while the same object written
through CreateEntry got the rule's TTL. Guard the shared stamping so the
two paths cannot drift apart again.
* filer: apply the path's storage rule to an appended entry
AppendToEntry resolved the storage option from the path - so its chunks
land on a TTL volume under an fs.configure TTL rule - but never stamped
the rule's TTL on the entry it creates, leaving an entry that outlives
its data. Route it through applyStorageDefaultsToEntry, which now feeds
the entry's own TTL into the option so the placement an existing entry's
appended chunks get is unchanged.
* filer: apply the path's storage rule to a completed TUS upload
The PATCH path resolves the storage option from the target, so a TUS
upload into an fs.configure TTL prefix writes its chunks to a TTL volume,
but completion built the final entry with ttlSec 0 - the entry outlived
the data it pointed at. Stamp it through applyStorageDefaultsToEntry,
which also subsumes the hand-rolled read-only check and supplies the
rule's name-length limit.
* filer: apply the destination's storage option TTL to a copied entry
The copy handler re-uploads the source's chunks under the destination's
storage option, so a copy into an fs.configure TTL prefix already lands
its data on a TTL volume. The entry, though, carried the source's ttlSec
- 0 for a source outside the prefix, or the source's own TTL where the
two rules differ - so it never expired with the data it pointed at. Take
the TTL from the same option the chunks were placed with, after the
data-only copy has restored the destination's metadata.
* master: cap the reported capacity at what the disks hold
Statistics reported max volume count times the volume size limit, which is
how many volumes the cluster is allowed to place, not how much space it has.
A cluster given far more slots than its disks can fill reported a capacity it
could never reach -- 65536 slots at 30GB read as 1.9PB on a 460GB disk -- and
the number never moved, since writing data changes neither the slot count nor
the size limit.
The volume servers already report each filesystem's total and free bytes in
their heartbeats, so bound the answer by what they say is left.
* mount: keep the last known sizes when filer statistics fails
A failed Statistics call returned before df's answer was filled in, so a
mount whose filer or master was briefly unreachable reported an empty
filesystem rather than the sizes it already had.
* master: drop the disk ceiling when a volume server does not report
A cluster part way through an upgrade has volume servers that predate the disk
bytes in the heartbeat. Summing only the ones that answered left the quiet
server's free space out of the total, and the server holding the room is
exactly the one that could make the cluster read as full.
Answer with the disks only when every one of them reported.
* rust volume: fail closed on sorted-index failures and reconcile tier-up
Follow-ups to the .sdx sorted needle map (#10951):
- get() folded open/read failures into None, so an EIO, a torn .sdx, or a
failed pooled reopen answered reads with NotFound and let do_delete_request
acknowledge the delete as Ok(0) without writing a tombstone. It now returns
io::Result and every caller propagates; redb's get() had the same shape and
is fixed with it. is_file_unchanged cannot propagate, so it reports unknown
and logs rather than treating an unreadable index as proof of a change.
- A delete whose .idx append landed but whose .sdx mark failed left the map
still resolving the old live entry, so deleted content stayed readable until
a reload. The map now records the tombstone before touching .sdx and only
clears it once the mark lands; lookups consult that first and report the
needle deleted, which is what the next reload concludes anyway.
- Mode reconciliation ran one way. Entering remote mode made use_sorted_index()
true, which returned early, so a volume tiered while the server runs kept its
in-memory map and pinned .idx descriptor until restart — the RAM and fd win
never applied. It now reconciles in both directions.
- Tier-down dropped the remote reference before the fallible refresh, so a
failure left volume_info local, the remote backend attached, the .vif still
remote, and a retry reporting "already on local disk". The transition is
snapshotted and rolled back.
- The read-only fallback set no_write_or_delete but left no_write_can_delete,
so metrics and mode checks called the volume delete-capable while every
delete was refused.
* rust volume: count a sorted-map delete against the durable .idx append
The deletion counters sat after the in-place .sdx mark, so a mark that
failed left them at their pre-delete values while the tombstone was already
durable in .idx — and with retries now idempotent, nothing applied them
later either. Heartbeats, status responses, and the garbage calculation
would report the volume as free of that garbage until a reload.
Move them to the append that makes the delete durable, which is also what a
reload of .idx would count. Covered by a test that injects a mark failure
through a cfg(test) seam: no portable filesystem trick reproduces it, since
a read-only .sdx fails the borrow long before the mark.
* rust volume: hide a pending tombstone from the sorted-map scans too
The overlay that keeps a needle deleted after a failed .sdx mark was only
consulted by get(). visit_live_entries still read the stale valid record
straight off .sdx, so ascending_visit, iter_entries and save_to_idx all
reported the needle live — and compaction takes iter_entries for the
complete live set, so it would copy the deleted content forward and
save_to_idx would write it back into the rebuilt .idx as live.
Snapshot the overlay once per scan and skip its keys, which is the same
conclusion the next reload reaches from the .idx tombstone.
* rust volume: quarantine a durable write whose index lookup fails
The prior-mapping lookup that decides whether to index a fresh append runs
after the record is already down and flushed, so a failing lookup leaves
exactly the state a failing put leaves: a durable .dat record nothing
indexes. The put path marks the volume read only for it; this one returned
the error and kept taking writes, and the next append would bury the
orphan mid-file where the .dat tail check on reload cannot see it.
Give it the same treatment.
* mount: name the disk after the mounted path
Finder and Explorer labelled every mount with the filer address, so two
mounts from one filer were indistinguishable. Use the mounted path's last
segment, the way df already shows it, and keep the filer address only for
a whole-tree mount.
* mount: let a given mount option override the default
The options from -o were placed before the ones this mount derives, so
a volname or iosize given on the command line lost to the derived value.
Append them last, matching the Windows adapter.
* mount: document what labels the disk
* worker: move the collection filter parser into weed/util/wildcard
The parser sits beside the volume-list filtering it was written for, in
weed/plugin/worker, which imports weed/shell — so the shell commands that
parse the same filter three other ways can never call it. Move it down to
weed/util/wildcard, next to the comma-separated wildcard helper it already
replaced, leaving the behavior unchanged.
* shell: parse every collection filter the same way
The shell parsed a collection filter three ways: compileCollectionPattern
compiled one regex for ec.encode, ec.decode, volume.balance and the tier
commands; volume.list and volume.deleteEmpty matched a single wildcard; and
volume.tier.move, volume.fix.replication and volume.configure.replication
called filepath.Match on their own. None of them took a list, so
"ec.encode -collection=a,b" selected nothing, the same way the admin UI did.
They all go through the shared matcher now: a comma-separated list of names,
"*" and "?" wildcards, "_default" for the collection with no name, and regex
entries. The one thing that stays per-command is what an empty value means -
every collection for -collectionPattern, the unnamed collection for the ec
and tier -collection flag - so compileCollectionPattern keeps that mapping.
The matchers are compiled once per command instead of once per volume, and a
regex entry now has to match the whole name unless it anchors itself, so
-collection=bucket no longer picks up mybucket2.
* shell: keep dots in collection names, and commas inside a regex
A dot no longer marks an entry as a regex, so a collection named "my.bucket"
matches itself and not "my-bucket" - the difference decides which volumes
volume.deleteEmpty and volume.tier.move touch. A dot still counts when it is
quantified, so "bucket.*" stays a prefix regex.
The comma split also leaves alone the commas inside a character class or a
repetition count, so "bucket[0-9]{1,3}" stays one entry instead of becoming
two broken fragments.
* shell: let a regex entry match its own spelling
A collection named after regex syntax, say "logs(2024)", was unreachable:
the entry compiled to a pattern that matches "logs2024" instead. Match the
entry verbatim as well, so naming a collection always selects it, whatever
characters it holds.
* shell: reject a collection filter that names no collection
A value of "," parsed to no entries and then matched every collection, so a
typo widened ec.encode or volume.deleteEmpty to the whole cluster. Only a
genuinely empty filter means "all collections"; anything else has to name one.
* shell: keep commas inside a regex group out of the entry split
The split already left alone the commas inside a character class or a
repetition count, but not the ones inside a group, so "bucket(foo,bar)"
was cut into two fragments that no longer compile.
* shell: cover escaping a collection name that is not a regex
A name like "logs(2024" does not parse as a regex on its own; escaping it,
"logs\(2024", reaches it. Pin that so the escape hatch does not regress.
* shell: split entries only on commas inside a closed regex construct
An unmatched "{" or "[" made the splitter swallow every comma after it, so
"foo{bar,videos" became one entry that matches neither collection - the
silent no-op this filter work exists to remove. A construct now has to close
before its commas stop separating entries.
* shell: skip character classes while scanning a regex group
A ")" inside a class is a literal, so "(a[)],b)" ended its group early and
split into two fragments that no longer compile.
* shell: cover escaping a comma inside a collection name
A comma separates entries, so a name holding one is reached by escaping it.
* shell: follow the regexp parser when scanning a character class
A "]" leading a class is a member of it, and a POSIX class such as
"[:alpha:]" carries its own "]", so stopping at the first one cut a valid
filter like "(a[]),],b)" into fragments and rejected it.
* s3: deny anonymous requests when the identity config loads no identities
Naming a config file is the operator asking for authentication. A file that
yields no identity - an unpopulated secret mount, or a mistyped top-level key
the proto parser silently drops - left the gateway open to every anonymous
caller: ListBuckets returned 200, and anonymous PUT could create buckets and
write objects.
* s3: name the unknown top-level keys in an identity config
The proto parser discards what it does not recognise, so a mistyped
"identites" loads as an empty config. Naming the dropped keys at startup turns
the resulting lockout into a one-line diagnosis.
* s3: isolate the auth-enforcement tests from AWS environment credentials
* s3: use a singular "identity" as the unrecognised-key example
Codespell rejects the misspelling the example used.
* s3: cover the empty identity config alongside the unrecognised key
* s3: cover a config file whose body is an empty object
* rust volume: search .sdx for read-only volumes instead of holding the index
The Go volume server loads every read-only volume through SortedFileNeedleMap:
the index lives on disk as a sorted .sdx, a lookup is a binary search, and
since #10950 no descriptor is held between lookups. The Rust server had no
counterpart. Read-only volumes built a full in-memory CompactNeedleMap, and
cloud-tiered ones — noWriteCanDelete, so not the read-only branch — went
through the writable path and pinned an .idx append handle on top of it. At the
hundreds of thousands of tiered volumes a real server carries, that is an index
in RAM and a descriptor each, for volumes nobody reads.
Port the sorted map and the bounded handle pool. A tiered volume now costs zero
descriptors and zero index bytes when idle; the pool keeps the hot handles open
so a busy volume does not pay an open() per needle. Handles are Arc<File>, so
an eviction cannot close one a reader still holds.
The generated .sdx is byte-identical to Go's — same sort, same last-write-wins,
same dropped tombstones — so a volume moved between a Go and a Rust server reads
whichever copy is already on disk. A test pins the bytes against a Go-generated
fixture.
* rust volume: fail compaction on an unreadable .sdx, and rebuild the map on tier-down
Two ways the sorted map could lose data.
iter_entries swallowed read errors and returned however many entries it managed
to collect. Compaction takes that vector for the complete live set, so a
truncated .sdx or a mid-scan I/O fault would commit a volume missing every
needle past the failure. Return a Result instead and abort. redb's
collect_entries dropped errors the same way on the same path, so it goes with
it.
Tier-down clears the remote mode and publishes the volume as writable, but the
map it booted with is the read-only sorted one. Its put always fails, so the
first write would append to the local .dat and then fail to index it, leaving
bytes nothing references — and a non-fsync write repeats it. Fold the
reopen_idx_for_write swap into refresh_remote_write_mode so the map always
matches the mode it just published; a rebuild that fails pins the volume
read-only rather than letting it take writes it cannot record.
Go reaches neither: its tier-down leaves noWriteCanDelete set, so the volume
stays read-only until a reload or an explicit mark-writable, which already goes
through reopenIdxForWrite.
* rust volume: keep read-only volumes mountable on a read-only index dir, and batch the .sdx scan
Building .sdx writes to the index directory, and load_index_sorted_file also
created a missing .idx there. A volume whose index sits on a read-only mount
took both paths and failed to load, where before it mounted read-only off an
in-memory index and served reads. Create the .idx only where deletes are
allowed, and fall back to the in-memory map when the sorted one cannot be
built, so a directory nobody can write costs memory rather than availability.
The end-to-end scan behind iter_entries, ascending_visit and save_to_idx read
one entry per syscall. Read 1024 at a time instead, the batch size
idx::walk_index_file uses. Positional reads, not a cursor: the handle is shared
with any other borrower.
Also gate the Go byte-parity fixture on the 5bytes feature it describes, which
is otherwise dead code in a 4-byte-offset build.
* rust volume: roll back a failed writable mark, and rebuild a torn .sdx
set_writable clears the read-only flags before it can know the rest will
succeed, but only the map rebuild rolled them back. An .idx writer that fails to
attach left the volume advertising writable over a needle map with no writer, so
puts landed in memory and were gone after a restart — the exact failure the
function exists to prevent. The read-only-mount fallback made it reachable: that
path loads an in-memory map with no writer attached. All three steps now run
behind one rollback point.
A .sdx whose length is not a whole number of entries was accepted as long as it
looked fresh, and truncation is what makes it look fresh. The entry count then
floored, hiding the last needle from lookups and from compaction, which would
commit the shorter set. Treat a torn file like a stale one and rebuild it from
.idx. Go writes .sdx in place rather than through a temporary, so a crash
mid-generation is a real way to produce one.
Appends now start at the last whole .idx entry too, so a torn tail there is
overwritten by the next tombstone instead of misaligning every row after it.
* rust volume: trim a torn .idx before writing to it, keep delete-only volumes online, count sorted-map deletes
Three from review.
Flooring the sorted map's append offset only protected its own positional
writes. Every writable path appends at EOF instead, so a partial row left by a
short write pushed the next row off alignment and the following load parsed the
rest of the file as garbage. Drop the partial row before attaching any writable
index writer — it is unrecoverable anyway, and every loader already skips it.
Go refuses to load such a volume at all; trimming keeps it mountable with the
rows before the tear intact.
The unwritable-index-dir fallback stopped one step short for volumes that allow
deletes, which is every tiered one: the in-memory loader opens .idx read-write
there and fails on the same directory that just refused the .sdx, so the volume
stayed offline. Give up the deletes instead — without a writer no tombstone
could be recorded anyway — and a remount on a writable directory restores them.
Sorted-map deletes left the counters untouched, so a tiered volume reported
itself garbage-free until it restarted. They now land where a reload would put
them: the tombstone is another .idx row, and both it and the row it supersedes
count as deletions under the rule the load-time metric applies. Go skips this
too, and should not.
The collection filter was parsed twice with two syntaxes: the master-side
volume listing compiled the whole string as one regex, while EC encode and
EC balance detection split it on commas and matched each entry as a
wildcard. A volume had to pass both, so "collection-a,collection-b" matched
nothing (no collection is named that), and the ALL_COLLECTIONS sentinel,
which the master side skips, dropped every volume at the task side.
Parse it once, in one place: a comma-separated list where an entry is a
name with optional * and ? wildcards, or a regex when it carries regex
syntax. A regex entry now has to match the whole name unless it anchors
itself, so listing a collection no longer picks up its longer namesakes.
* filter volume by local or remote storage name
Signed-off-by: lou <alex1988@outlook.com>
* fix SelectsEverything
Signed-off-by: lou <alex1988@outlook.com>
* keep the proto sync out of this change
The branch copied weed/pb/*.proto over their seaweed-volume and Java
counterparts and regenerated every .pb.go with a different protoc and
protoc-gen-go-grpc. DiskStatus.error arriving that way broke the Rust
build, and the rest is toolchain churn in files this change has nothing
to say about.
---------
Signed-off-by: lou <alex1988@outlook.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
A read-only or cloud-tiered volume loads a SortedFileNeedleMap, which held
both its .idx and its .sdx open for the life of the process. On a server with
~600K tiered volumes that is 1.2M descriptors before a single read, enough to
exhaust the fd limit and take the listeners down. The .dat is not the problem:
a tiered volume serves it from the remote backend.
Neither index file is needed except while a lookup is in flight, so borrow them
from a bounded process-wide pool instead. An idle volume now holds zero
descriptors; a busy one keeps its handles hot rather than paying an open() per
needle. Reads borrow O_RDONLY, so a volume on a read-only mount answers lookups
that previously failed at load. Sync tracks whether a tombstone was appended,
which also drops the fsync-per-volume storm at shutdown.
* master: estimate a hintless assign's size from the volume's average file size
An assign that carries no dataSize hint charged a flat 1MB per file id
against the volume's effective size. A small-file workload overpays by
orders of magnitude: bulk-writing 4KB files marks volumes holding a few
hundred MB of real data as crowded and then full, so the master grows
unnecessary volumes and, once every volume is spuriously full, fails all
assigns. Estimate from the volume's own average file size instead, and
keep the 1MB fallback only for volumes with no history.
* master: decay pending assign sizes for volumes gone quiet
The decay that corrects pending assign estimates runs only when a
heartbeat reports the volume, and a heartbeat only reports a volume
whose content changed. A volume held out of the writable list takes no
writes, so once inflated estimates mark every volume full, nothing is
ever reported again, nothing decays, and the cluster refuses all writes
until a restart. Run the decay from the master's periodic loop for
volumes no heartbeat has reported within two pulses, feeding the last
reported size back through the same path an unchanged heartbeat would
take.
* master: trim the comments on the assign size estimate
* master: keep the periodic decay out of the replica-dedup window
UpdateVolumeSize ignores a report arriving within two seconds of the last
one, so replicas of the same volume do not each halve the pending
estimate. The periodic decay went through the same path and stamped that
window, so a real heartbeat landing right behind it was dropped along
with its reported size and compact revision. Only a volume whose content
changed is reported at all, so nothing would send that size again and
the master kept a stale one. Let the dedup window belong to volume
server reports alone.
* master: let the decay read the size record under the lock it mutates
The periodic decay picked its volumes under a read lock and replayed
them under a write one, carrying the size it had read across the gap. A
heartbeat landing in between was rolled back: the replay wrote the older
size and compact revision over the fresh ones, and a compaction report
lost that way is never resent, since only a volume whose content changed
is reported. The decay has no size of its own to contribute, so it now
reads the record under the same lock it mutates.
* master: let a heartbeat that beat the decay stand for the cycle
The decay chooses its volumes under a read lock and applies them under a
write one. A heartbeat landing in that gap already did the halving the
cycle owed, so applying the decay on top of it halved twice and forgot
pending bytes the volume has not written yet - the double-halving the
replica-dedup window exists to prevent. Both callers now give way to a
report already handled for this cycle; only a real report still advances
lastUpdateTime, so a quiet volume keeps decaying every pulse.
* master: keep genuinely full volumes out of the decay pass
A volume the disk really did fill keeps its fullSince set for good, so it
was selected every pulse for a decay that cannot help it: UpdateVolumeSize
refuses to recover a volume whose reported size is at the limit, and
replaying a size that cannot move leaves the record as it found it. Full
and quiet is the ordinary resting state of a cluster, so this was most of
the pass, taking the layout write lock away from the heartbeats to do
nothing. On a million tracked volumes with a hundredth of them phantom-full
it costs ten thousand write locks a pulse instead of a million.
* master: put the stale-replay test back on the path it guards
Giving the decay the dedup window left this test short-circuiting there,
so it no longer reached the locked read it was written for and passed
with that read removed. Age the record past the window, which is the only
case where reading it under the lock is what saves the report.
* grpc: a non-cancellable context is no evidence of a stale channel
shouldInvalidateConnection only invalidates on Canceled/DeadlineExceeded
while the context handed to WithGrpcClient is still live, so that an RPC
timing out on its own does not close the shared cached ClientConn and
cancel every other in-flight RPC on it. context.Background()/TODO never
expire, so Err() stays nil forever and that guard always answered
"invalidate" - and Background is what almost every caller passes, the S3
gateway included.
One S3 request whose RPC rode an abandoned HTTP request context therefore
closed the shared filer connection, and every multipart part in flight
died with "the client connection is closing", surfacing to the client as
400 InvalidRequest.
Only a cancellable context bounds an RPC attempt, so require one before
reading it. A genuinely stale channel (a peer restart behind a stable L4
endpoint) surfaces as Unavailable, which invalidates on its own branch.
* grpc: a bystander of a connection teardown is not a stale-channel witness
gRPC raises ErrClientConnClosing locally, before an RPC reaches the wire,
when this process has already closed the ClientConn. Every caller that
touches a channel during another goroutine's teardown gets it, so reading
it as a stale-channel signal lets one teardown re-arm itself across the
whole herd of callers it just cancelled.
The cached-connection version check keeps those callers from closing a
replacement channel, but the streaming path invalidates by address alone
and has no such guard.
* grpc: end a stream without dropping the peer connection under it
A streaming caller gets its own ClientConn, but on any error it also drops
the cached non-streaming ClientConn every request handler shares with that
peer, to recover a peer restart hidden behind a stable L4 endpoint. Any
error includes the ordinary ones: a metadata subscription that reached its
stop point, a follow callback that refused an event, a caller that gave up.
The S3 gateway follows filer metadata on such a stream and reconnects
forever, so each ordinary end of it cancelled every S3 request in flight
against the filer. Drop the shared channel only for errors that say the
peer went away, which is what invalidation is for.
* test: close the connections the cascade tests leave cached
Each test swaps in a fresh connection cache and restores the previous one,
dropping its own entries without closing them, so the ClientConn's
transport and reconnect goroutines outlive the fake filer they dialed.
* grpc: say why ErrClientConnClosing's deprecation notice does not apply
It points at codes.Canceled, which is the code this function exists to
disambiguate. Only the message distinguishes a teardown a caller merely
walked into, so the sentinel stays.
* filer: store TUS sub-chunks through the regular chunk writer
A TUS sub-chunk was written with one assigned file id, retried up to
three times against that same id, and abandoned on failure: an attempt
that had landed on some replicas left a needle no session record and no
entry ever references, unreclaimable by vacuum.
dataToChunkWithSSE, which the regular write path uses per chunk, assigns
a fresh file id per attempt and hands back the file ids of failed
attempts, which are now freed the way the regular write path frees them.
* filer: retry a chunk write on a fresh volume when the server 5xxs
The filer's chunk writer assigns a fresh file id per attempt but only
retried transient network errors, so a volume filling up and turning
read-only mid-write failed the whole request even though the very next
assignment would have landed elsewhere. Every other write client already
routes this through ShouldReassignUpload; the filer's own write path now
does the same, for regular uploads and TUS sub-chunks alike.
* filer: export the chunk deletion queue
The filer test harness in weed/server builds filer.Filer as a struct
literal, so any code path reaching DeleteChunks dereferenced a nil
queue. Exported like the neighboring DeletionRetryQueue so the harness
can arm it.
* filer: complete a TUS upload whose chunk records overlap
A PATCH retried while its predecessor was still storing a sub-chunk -
a proxy timeout with an immediate retry is enough - records the same
range twice. HEAD computes Upload-Offset as the covered watermark and
reported the upload fully received, but completion demanded exactly
adjacent records and failed every attempt: the client concluded success
from offset == length, no entry was created, and the session eventually
expired, turning the entire upload into deleted needles for the vacuum
to chew through.
Completion now validates gapless coverage with the same watermark HEAD
uses. A record extending coverage joins the entry - the read path
resolves partial overlaps by ModifiedTsNs, and the raced copies carry
identical bytes - while a fully covered duplicate is freed once the
entry lands.
* filer: allow one mutating TUS request per session at a time
Nothing stopped two PATCHes from writing the same range concurrently:
both loaded the same offset, both passed the conflict check, and both
recorded their sub-chunks. A client whose request timed out in a proxy
retries immediately while the server side is still storing the buffered
sub-chunk, which is exactly that race.
A session now accepts one PATCH or DELETE at a time, the way tusd locks
uploads; a concurrent one is refused with 423 Locked, which TUS clients
retry, and HEAD keeps answering so progress polling is unaffected. The
chunk state is loaded under the claim, so a retried PATCH sees every
record its predecessor left and conflicts cleanly instead of duplicating
data.
* test: cover a TUS PATCH raced by its own retry
Stalls a PATCH mid-body over a raw connection, retries the same range
while it is in flight, and expects the retry refused with 423 Locked;
the upload then resumes from the reported offset and the final content
must be intact.
* filer: never free a TUS duplicate the entry still references
Coverage is computed from ranges, so a record fully covered by another
is treated as a duplicate no matter which needle it names. A malformed
record naming a file id the entry keeps would have had that needle freed
right after the entry landed - the corruption this change set exists to
stop. The duplicates are now freed in one batch, skipping any file id
the entry references; their records go with the session directory.
* test: bound the raw TUS connection reads
http.ReadResponse on the stalled PATCH's connection blocked until the
whole go test timeout if the filer never answered.
* filer: free the needles of chunk write attempts a retry replaced
A volume server stores the needle locally and only then fans out to the
replicas, so a replication failure 5xxs with the data already written.
Each attempt assigns its own file id, so once a later attempt lands
elsewhere nothing references the earlier ones: the caller only sees the
chunk that succeeded, and the failed ids were dropped.
They are now freed the way the caller frees them when the whole write
fails. Retrying on a 5xx makes this reachable on every read-only or full
volume, which is exactly the condition that filled the reporter's
volumes.
A \\server\share -dir was passed to WinFsp as a plain mount point, which
treats it as a directory path on an actual remote server and fails. Turn it
into the VolumePrefix option instead, so the mount registers with the WinFsp
network provider: the UNC path is then reachable from every logon session,
which a drive letter mounted from a service is not, and each user can map
their own drive letter to it.
Comments only, no rendering change: the values paragraphs compress to
the density of the file around them, the env-var note becomes a
template comment instead of leaking into the rendered manifest, and the
two spots that invite a wrong simplification - the unconditionally
rendered -port.lance and the empty-placeholder platform guard - each
get their one-line why.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* helm: serve the S3 gateway's Lance Namespace, on by default
Standalone `weed s3` serves the Lance Namespace API on 9101 unless told
not to, so the chart defaulting s3.lancePort to 9101 matches weed's own
posture instead of hiding the port behind a null. The flag is always
rendered, so lancePort: 0 reaches weed as -port.lance=0 and genuinely
disables the namespace rather than silently falling back to the binary
default; 0 also drops the service port and the optional lanceIngress,
which otherwise mirror the iceberg wiring. The NetworkPolicy admits the
port the same way it admits icebergPort.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* helm: run the Lance maintenance worker beside the Go worker
The Go and Rust workers have no overlapping jobs - Go serves vacuum,
balance, EC and iceberg_maintenance, only /usr/bin/weed-worker serves
the lance_* family - so a cluster serving Lance tables needs both, not
an either/or switch. The worker deployment now adds a worker-lance
container whenever the namespace is reachable: worker.namespaceUrl, or
derived from the release's S3 service and s3.lancePort. Untouched Go
container; admin address derived the same way; mTLS flags point at the
already-mounted worker cert when security is on; metrics on their own
worker.lanceMetricsPort (9328, next in the 932x convention) with the
same health probes, service port and scrape endpoint the Go container
gets, and the worker NetworkPolicy admits that port exactly when the
container renders. The image carries an empty placeholder on armv7/386
where exec falls back to the shell and exits 0, so the command refuses
those platforms by name; s3.lancePort: 0 is the escape hatch there.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* filer.backup: key the checkpoint by source path and sink destination
The checkpoint id hashed only sink name + directory, so two backups to
different buckets or endpoints sharing a directory layout advanced one
checkpoint: whichever job was running pushed the shared offset forward,
and a stopped or failing job later resumed from the other's position,
silently skipping changes. Backups of different source paths to the same
destination shared a checkpoint the same way.
Each sink now reports a destination identity (endpoint or account,
bucket or container, directory) and the checkpoint is keyed by the
source path plus that identity. Reads fall back to the historical
name+directory key when the new key has no value, so existing backups
resume where they left off; writes go only to the new key.
* filer.sync: include the target path in the offset key
The offset stored on the target filer was keyed by source path and
source filer signature only, so two syncs from the same source cluster
and path to different directories on the same target cluster advanced
one shared checkpoint, and the slower one could resume past events it
never applied. The target path now participates in the key; "/" keeps
the historical form, and a sync with a non-root target path falls back
to the historical key once when its own key has no value yet.
* join checkpoint key fields with NUL so they cannot alias
A path or configuration value spelling out the separator could
concatenate two different field tuples to the same checkpoint key.
NUL cannot appear in a CLI path argument or any sane configuration
value, making the encoding injective.
* admin: offer copyable S3 object URLs in the bucket file browser
* admin: hide object urls when the bucket type lookup fails
* admin: ignore an s3.public_endpoint that is not an absolute http url
* mini: build the seeded s3 endpoint with JoinHostPort for ipv6
* admin: reject a query or fragment in s3.public_endpoint
* mini: drop the seeded s3 endpoint when a later run disables s3
* admin: reject userinfo and bare delimiters in s3.public_endpoint, redact the warning
* mini: pass its s3 endpoint as an admin option instead of mutating viper
* admin: keep the rejected s3.public_endpoint value out of the log
The SSE streaming path kept writing 200/206 from filer metadata before
fetching or decrypting anything, so a missing needle or failed decrypt
setup surfaced as a broken 200 body. Same deferral as the plain path:
the status commits on the first body write, and every failure before
that returns to the handler for a clean S3 error response.
streamFromVolumeServers wrote the 200/206 status from filer metadata
before any byte had been fetched from a volume server, so a missing or
corrupted needle surfaced as a broken 200 body and the request metrics
recorded a success. Defer the status commit to the first body write: a
failed first read now returns a clean 500 before headers, while the
wire timing of successful responses is unchanged since net/http buffers
the status line until body bytes arrive anyway.
* filer: keep the sentinel when CreateEntry reports an update failure
CreateEntry flattened the error UpdateEntry wraps, so errors.Is stopped
matching and ErrExistingIsDirectory and ErrExistingIsFile never reached
the S3 mapper, which answered a retryable 500 instead.
* s3: let a key that is a prefix of other keys be an object
S3 keys are flat, so "a/b" and "a/b/c" are independent objects that
coexist in either write order. The filer stores a key as a path, so one
of them has to live on the directory the other is nested under.
Writing the nested key first refused the prefix key outright. Writing it
second promoted the file to a directory, which kept its data but lost the
key: an empty object left nothing to recognise it by and disappeared, and
one with data listed under a trailing slash it never had.
Mark the directory that carries such a key, and write the object onto it
when the path is already a directory. The mark makes an empty prefix
object visible to listings and readable by GET and HEAD, keeps the empty
folder cleaner off it, and lists it under the key it was written with.
Deleting the key strips the mark back off along with the data.
* filer: keep a TTL off a directory that stands for an object
An expired entry is deleted a row at a time, so expiring a directory
removes it and leaves everything under it unreachable. Promoting a file
to a directory carried its TTL across, and a promoted file is exactly the
one that has keys nested under it.
Drop the TTL on promotion, and leave one an older build wrote alone. The
lifecycle worker still expires the object, through the delete that leaves
the directory behind.
* s3: delete the null version of a key other keys are nested under
The routed delete cannot remove an entry that other keys live under, and
answered a retryable 500 rather than falling back to the lock path the
unversioned delete already falls back to. That path then looked the entry
up under the bucket with the whole key as its name, so the demote wrote it
back one directory too high and failed as not found.
Fall back on any non-precondition error, and split the key before deleting
it. Trailing-slash directory markers with children reach the same delete.
* filer: keep the sentinel when MkFile and Mkdir report a create failure
Same flattening one layer out: every mkFile caller lost the sentinel, so
a CopyObject onto a key that other keys are nested under answered a
retryable 500 where a PutObject of the same key answers 409.
* s3: copy and rename a key that other keys are nested under
Such a key is stored on the directory those keys live in, and copy and
rename both refused it: the source lookup maps every directory entry to
NoSuchKey, so a key a plain GET serves could not be copied or moved, and
the destination side refused it as a directory conflict.
The source is read through a view of the entry as the object it names.
The destination is written the way a PutObject of that key writes it. A
rename at either end copies the object's own data across and strips it off
the source key rather than going through AtomicRenameEntry, which moves a
directory by moving everything under it - the nested keys are not part of
what is being renamed.
* mini: reserve the admin gRPC port instead of binding it late
Port selection probes every port with a throwaway listener and closes it.
Master, filer, volume and S3 bind a moment later, but the admin waits for
all of them first and only then binds its worker gRPC port, roughly two
seconds in. That port defaults to the admin http port + 10000, which lands
inside the Linux ephemeral range, so one of the cluster's own outgoing gRPC
dials can take it during the gap and the admin dies on bind, taking the
worker with it.
Keep the listener from the availability check and hand it to the admin.
* mini: clear the admin gRPC reservation before retaking it
A rerun inside one process would otherwise inherit the closed listener of
the previous run whenever the reservation fails, and the admin would accept
it and only find out inside Serve.
* mini: snapshot the admin options for the startup goroutine
The cleanup path read the package-level options long after the goroutine
started, so a later in-process run could have its reserved listener closed
by the previous run.
* ec: read a needle's intervals in parallel
A needle spanning more than one EC block gets one interval per block, and
consecutive blocks live on different shards. We read those intervals in
sequence, so a 4MB chunk landing in a volume's 1MB small-block region cost
five round trips to five different servers.
Read them concurrently into disjoint slices of a single buffer, at most 8 in
flight. Same change in the Rust volume server's phase C.
* ec test: seed the random payload instead of the deprecated rand.Read
* admin: treat a missing S3 Tables policy as an empty load, not an error
The bucket/table policy GET relayed the backend's 404 NoSuchPolicy to the
dialog, whose loader treats any non-OK response as a load failure and
keeps Save and Delete blocked. A bucket or table without a policy could
never be given one. Return policy null instead, the same contract
ShowBucketPolicy uses for classic buckets.
* admin: reject policy documents the structured editor would misread
A top-level JSON array passed the object guard (typeof [] is 'object')
and loaded as a zero-statement policy, which the next commit would
rewrite to an empty document. Object elements in Action/Resource were
coerced to '[object Object]' and saved that way on the s3tables surface,
which stores policies verbatim. Both now throw, which routes the
document to the JSON tab like other unrepresentable shapes.
* admin: let the JSON tab save documents the structured editor can't model
Save with the JSON tab active required a round-trip through
policyDocToEditorState, so exactly the documents the dialogs shunt to
'JSON tab only' mode (unrepresentable Effect, Resource+NotResource, and
the like) could never be saved - Delete was the only mutation left.
Invalid JSON still blocks; an unrepresentable document now saves and the
editor state stays marked unparsed.
* admin: pin the policy editor to what each consumer's backend supports
The s3tables evaluator has no NotResource/NotPrincipal fields - it
silently drops them, turning Allow+NotResource into allow-everything and
making Deny+NotPrincipal inert - and it only matches s3tables: actions
against s3tables ARNs, while the editor suggested s3: actions and
arn:aws:s3::: resources. New registerPolicyEditor knobs: allowNegation
hides the Not* modes and routes documents using them to the JSON tab;
resourceSuggestions pins the Resource autocomplete to the open
resource's ARN; the S3 Tables dialogs get an s3tables-only action
datalist. requirePrincipal now also hides NotPrincipal, which
policy_engine.ValidateBucketPolicy always rejects, and the client-side
check requires Principal specifically to match that server rule.
* admin: save S3 Tables policies from a button, not form submission
The multi-input structured editor sits inside a form whose Save button
was type=submit, so Enter in any single-line editor input - accepting an
autocomplete suggestion, say - implicitly submitted whatever half-built
statement the editor held, and the backend stores the document verbatim.
A lone statement with no Principal matches nobody, locking out every
non-owner. Save is now an ordinary button and the form ignores
submission.
* admin: block zero-statement policy saves
Committing the active tab before the emptiness check made 'Policy JSON
is required' dead code: an empty editor serializes to {"Statement":[]},
which the s3tables backend stores verbatim - evaluated default-deny for
every non-owner, while the statement-count column keeps showing 'Not
configured'. All three policy dialogs now refuse a save with no
statements and point at Delete instead. The classic bucket modal only
gained a clearer message; the server already rejected the document.
* admin: guard S3 Tables policy mutations against stale and overlapping requests
The save/delete completions ran against whatever resource the shared
modal happened to show by then: a slow PUT for one bucket would hide the
modal mid-edit of another and misattribute its alerts, a late DELETE
cleared the shared textarea over the newly opened resource with its
loaded flag set, and nothing stopped a double-click from firing two
overlapping mutations. Ported the classic modal's pattern: capture the
target on start, flag the mutation in flight with the buttons disabled,
and only touch the UI when the completion still matches the open
resource. Success now reloads the page, which also keeps the Policy
column's statement count honest.
* admin: confirm before deleting an S3 Tables policy
Delete Policy sat next to Save and fired on a single click; with
default-allow enabled one stray click silently dropped the resource
policy and left the bucket open to every principal. Same confirmation
the classic bucket modal already has.
* admin: let a corrupt stored bucket policy be shown, fixed, and deleted
A stored document the decoder rejects made the policy GET 500, and with
the loaded flag never set the modal blocked both Save and Delete - the
one policy an operator most needs to remove was the one they couldn't,
even though the delete path never reads the document. The GET now
returns the raw bytes alongside a null policy; the dialog hands them to
the JSON tab and unblocks the buttons.
* admin: url-encode the bucket name in the policy API calls
The filer lists any directory under the buckets path, names S3 would
never allow included; one carrying '#' or '%' broke the fetch URL or
addressed a different name than the modal shows.
* admin: drop stale edit-policy responses on the IAM policies page
The same race the bucket and S3 Tables dialogs already guard against:
open one policy's editor while its GET stalls, open another, and the
late response populates the editor under the second policy's name -
Update then saves the first policy's statements over the second.
* admin: warn before a bucket policy save drops unsupported fields
The editor tracks unmodeled top-level keys precisely so
confirmPolicyFieldDiscard can warn before the server's Version+Statement
decode discards them, but only the IAM page called it; the bucket modal
saved a pasted document with e.g. a console-generated Id without a word
while the editor kept displaying the field.
* s3: enforce the bucket policy size cap on both surfaces
The 20KB cap lived only in the admin UI, so a larger policy stored via
the S3 API displayed there but could never be re-saved, desyncing the
two writers the cap comment claimed could not desync. The constant now
lives in policy_engine next to the shared validator and PutBucketPolicy
rejects oversized documents with PolicyTooLarge, matching AWS.
* admin: ship the policy editor's fieldset styles with the editor
The .policy-stmt-* rules that undo Bootstrap's full-width legend reset
stayed behind in policies.templ when the editor markup moved to the
shared script, so the bucket and S3 Tables dialogs rendered Actions/
Resource/Principal as full-width jumbo headings. PolicyDatalists is the
component every consumer already renders once; the styles live there
now.
* s3: mirror bucket policy changes into the IAM store from the metadata subscription
The advanced-IAM path appends the bucket-policy:<bucket> document to
every STS/session evaluation, but only this gateway's own PutBucketPolicy
maintained that mirror - a policy tightened or created through the admin
UI (or another gateway) never reached it, so revoked access stayed live
indefinitely, and the delete side was an unimplemented TODO in any case.
The metadata subscription now diffs the stored policy on every bucket
entry change and updates or removes the mirror, covering all writers and
deletion with one mechanism; IAMManager gains the missing
RemoveBucketPolicy.
* admin: deduplicate the bucket policy write path
Set and Delete carried line-for-line identical filer closures;
bucketPolicyMutation already treats nil as clear-the-key. The shared
helper sits below Set's validation, since ValidatePolicy cannot take the
nil document Delete passes.
* s3: drop ValidateBucketPolicy's re-checks of ValidatePolicy rules
Both callers run ValidatePolicy first, which already enforces the
version and at-least-one-statement rules; the duplicates were dead code
with drifted error text.
* admin: seed a new statement's Resource from the pinned suggestions
A fresh statement on the S3 Tables dialogs started with no resource row
at all; seed it with the broadest pinned ARN the same way cfg.bucket
already seeds the classic modal.
* admin: refuse to save Not* fields the backend would silently drop
Hiding the NotResource/NotPrincipal modes was not enough where negation
is disallowed: the JSON tab accepts any valid document (that is its
job), and a statement's Advanced-fields box can reintroduce the keys, so
an s3tables save could still store fields the evaluator drops - turning
Allow+NotResource into allow-everything. commitPolicyActiveTab now runs
a final document-level check over what would actually be saved; Delete
stays available for cleanup.
* s3: move the IAM bucket policy mirror on a bucket rename
A same-directory rename delivers one event carrying both entries, and
the byte-equality short-circuit skipped the new name's mirror when the
policy was unchanged - while the replayed delete for the old name
removed its mirror, leaving the renamed bucket unmirrored. The mirror
decision is now a pure function that removes the old name and writes the
new one regardless of byte equality, with the rename cases unit tested.
* s3: backfill the IAM bucket policy mirror on lazy bucket loads
The metadata subscription only mirrors changes, so a policy that
predates the IAM integration never reached the bucket-policy:<bucket>
mirror and its grants did not bind on the IAM path until the policy was
next modified. The gateway is deliberately lazy at startup (nothing
lists all buckets), so the backfill hooks the same place a bucket's
policy first becomes known: the cold bucket-config load. EnsureBucketPolicy
writes only when no mirror is stored, so repeat loads cost one cached
read.
* s3: reconcile the bucket policy backfill against concurrent changes
The backfill's check-then-write could race an event-driven mirror update
or removal and re-store bytes that were already stale, with no later
event to heal it. EnsureBucketPolicy now reports whether it wrote, and a
write is reconciled against a fresh authoritative entry read: a changed
policy is re-mirrored, a removed one is removed. Anything changing after
that read fires its own event, which finds the backfill's write already
present and supersedes it. The backfill also carries the entry's raw
bytes rather than a re-marshaled document, so the reconcile can
byte-compare.
* s3: prime the bucket policy mirror before advanced-IAM authorization
The backfill ran from the lazy bucket-config load, but IAM authorization
evaluates the bucket-policy:<bucket> mirror before any handler runs - a
grant carried only by a not-yet-mirrored policy denied forever, and the
denied request never reached the code that would have loaded the bucket.
authorizeWithIAM now primes the bucket config first (an in-memory cache
hit once warm), and the backfill runs synchronously on the cold load so
the very first authorization already sees the mirror.
* admin: honor a persisted or admin.toml maintenance enabled=false
The startup path discarded an operator's enabled=false twice over:
ApplyDefaultsToProtobuf treated the bool zero value as unset and applied
the schema default of true, and a force-enable migration block flipped
any survivor. With the legacy /maintenance UI routes gone, nothing could
write the config either, so the maintenance system ran unconditionally.
Keep the persisted enabled flag across schema-default application in
LoadMaintenanceConfig, drop the force-enable block, and add a top-level
[maintenance] enabled key to admin.toml as the config surface, persisted
through SaveMaintenanceConfig like the per-task settings. Absent config
still defaults to enabled.
* admin: track presence on the maintenance enabled flag
A plain proto3 bool cannot distinguish an operator's persisted false
from a legacy file that simply omits the field, so honoring false would
have silently switched maintenance off for configs written before the
toggle could be persisted. Make the field optional: files that predate
presence tracking keep the enabled default, while a file that explicitly
persists the toggle is honored either way.
* admin: manage S3 bucket policies from the admin UI
Bucket policies were only manageable through the S3 PutBucketPolicy API;
the admin UI had no equivalent to the quota/owner/lifecycle editors it
already offers. Add GET/PUT/DELETE for a bucket's policy, sharing the
exact validation the S3 gateway uses.
- Extract validateBucketPolicy/validateResourceForBucket out of
s3api_bucket_policy_handlers.go into policy_engine.ValidateBucketPolicy /
ResourceMatchesBucket so both the S3 API and the admin UI enforce
identical rules.
- weed/admin/dash/bucket_policy.go: Get/Set/DeleteBucketPolicy, writing
through ObjectTransaction + PATCH_EXTENDED (the lifecycle pattern) so a
concurrent owner/quota/lifecycle change on the same bucket entry isn't
clobbered. Propagation to every S3 gateway is automatic via the existing
filer metadata log subscription. The S3 gateway's IAM policy mirror is
deliberately not replicated here (its delete path is already an
unimplemented TODO on the S3 side).
- New GET/PUT/DELETE /api/s3/buckets/{bucket}/policy routes, CSRF-guarded
on writes.
- Bucket list and details modal now show a statement-count badge, read
from the entry already fetched (no extra RPC).
- UI: a JSON-textarea policy editor modal, matching the lifecycle modal's
structure.
* admin: reuse the visual policy editor for bucket policies
Extract the structured policy editor (add/remove statement, action/
resource/principal rows with autocomplete, JSON tab kept in sync) out of
policies.templ's inline script into a shared
weed/admin/static/js/policy_editor.js, and wire the bucket policy modal
in s3_buckets.templ up to it instead of a bare JSON textarea.
- registerPolicyEditor(which, config) replaces the hardcoded create/edit
id derivation with a per-instance config (textarea/tab/body ids,
datalist ids, requirePrincipal, bucket). The IAM policies page keeps its
exact pre-extraction ids via two registerPolicyEditor calls, so its
markup is unchanged.
- New policy_datalists.templ exposes the three shared <datalist>s
(actions/resources/principals) as @PolicyDatalists(), now rendered by
both policies.templ and s3_buckets.templ.
- requirePrincipal seeds new bucket-policy statements with Principal: "*"
and adds a client-side check before save (the server, via
policy_engine.ValidateBucketPolicy, remains the actual authority); the
bucket config pins the Resource autocomplete to the open bucket instead
of fetching every bucket in the cluster.
- layout.templ loads policy_editor.js globally, after admin.js/
modal-alerts.js (basePath/escapeHtml/showAlert) which it depends on.
3a (the extraction) is a byte-preserving move verified against the
unchanged policies.templ behavior before layering 3b's parameterization
and the bucket-policy wiring on top.
* admin: migrate S3 Tables bucket/table policy editors to the shared editor
Third consumer of the shared visual policy editor: the S3 Tables bucket
and table policy modals (a bare JSON textarea each) now get the same
structured Editor/JSON tabs as the bucket policy and IAM policy pages,
via registerPolicyEditor('s3tablesBucketPolicy'/'s3tablesTablePolicy',
{ textareaId: ... }). Storage and validation are untouched - S3 Tables
policies still go through their own s3tables.PolicyDocument type and the
s3tables.policy extended attribute, unrelated to policy_engine and
s3-bucket-policy; only the editor UI is shared.
Fix a real bug surfaced by adding this second load path: the bucket
policy modal (and the naive first draft of this s3tables port) called
commitPolicyTextareaToEditor() right after a GET and then force-switched
to the Editor tab. commitPolicyTextareaToEditor() is designed to leave
the current tab in place and the editor state untouched when a document
fails to parse (so an in-progress edit survives a bad tab switch), so
forcing the Editor tab afterwards could show empty/stale editor state
that a careless Save would then serialize over a perfectly valid but
structurally-unusual stored policy. Add
loadPolicyTextareaIntoEditor(which) to policy_editor.js, which has no
"current tab" to defer to and instead falls back to the JSON tab with an
alert on a document the structured editor can't represent - the same
safety editPolicy already had in policies.templ - and use it at all three
"populate the editor right after a GET" call sites (bucket policy,
S3 Tables bucket policy, S3 Tables table policy).
* admin: show policy statement count on the S3 Tables buckets page
Mirrors the "Policy" column already added to the classic S3 buckets
list: a clickable badge with the statement count when the table bucket
has a resource policy, "Not configured" otherwise. S3 Tables policies
are a separate mechanism (s3tables.PolicyDocument under the
s3tables.policy extended attribute) from the S3 bucket policy work
elsewhere in this branch (policy_engine.PolicyDocument /
s3-bucket-policy), so this is a parallel implementation of the same
pattern rather than shared code.
- S3TablesBucketSummary gains PolicyStatementCount, populated in
GetS3TablesBucketsData from entry.Entry.Extended[s3tables.ExtendedKeyPolicy]
via the new extractS3TablesPolicyStatementCountFromEntry - no extra RPC,
the entry is already fetched for ExtendedKeyMetadata.
- The badge reuses the existing .s3tables-bucket-policy-btn class, so it
opens the same policy modal as the row's action button with no JS
changes.
* admin: don't let a failed policy GET open the door to an empty overwrite
loadS3TablesBucketPolicy/loadS3TablesTablePolicy cleared the textarea,
then unconditionally called loadPolicyTextareaIntoEditor() regardless of
whether the GET actually succeeded - including when fetch() rejected or
the response was not ok, silently logged to console only. That leaves
the structured editor holding a legitimate-looking empty policy
({version, statements: []}), with the Editor tab active by default.
If Save is then clicked, commitPolicyActiveTab() serializes that empty
state into the textarea as `{"Version":"2012-10-17","Statement":[]}` -
a non-empty string - before the "Policy JSON is required" guard ever
sees it, so the guard passes and the transient load failure gets
written over whatever policy was actually stored.
Add s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded, set true only
once a GET has actually completed (ok, including a genuinely empty
policy) and false on any failure path (fetch rejection or a non-ok
response, which previously fell through silently). Both submit handlers
now check the flag before touching the editor at all, and a failed load
surfaces via alert() instead of only a console.error - the user
previously had no visible indication the load had failed.
Verified with a jsdom simulation driving the real rendered page against
a stubbed fetch: a failed GET followed by Save now sends no PUT at all
(previously it sent Statement: []); a successful GET followed by Save
still PUTs the loaded policy unchanged.
* admin: address code review findings on the policy editor
1. policy_editor.js: policyEditors is only pre-populated for 'create'/
'edit'; every other `which` (bucket, s3tablesBucket, s3tablesTable)
stays undefined until its first successful async load. Nothing in
this file enforces that a page hide its Editor/JSON tabs and
Add-statement button until that load completes - the S3 Tables policy
modals don't - so a click in that window (e.g. Add statement, or
switching to the JSON tab) threw "Cannot read properties of undefined
(reading 'unparsed')". Add policyEditorState(which), which lazily
initializes a default state, and route addPolicyStatement, the
jsonTabBtn 'show.bs.tab' handler, commitPolicyActiveTab, and
renderPolicyEditor through it. Verified with a jsdom simulation
against a never-resolving fetch: the exact click threw on the
pre-fix code and no longer does.
2. s3_buckets.templ: the bucket-policy Save handler checked the
textarea for emptiness before calling commitPolicyActiveTab(), which
is what actually serializes the structured Editor tab's fields into
that textarea. A policy entered entirely through the Editor tab (the
primary path - never touching the JSON tab) left the textarea at
whatever it was at load time, so creating a new policy this way hit
"Enter a policy document" and Save silently did nothing. Move the
commit before the emptiness check, preserving the existing alert and
early-return. Verified with a jsdom simulation: Add-statement then
Save (no tab switch) now PUTs the entered statement; before the fix
the same sequence never reached fetch().
3. s3tables_buckets.templ / s3tables_tables.templ: the policy Editor/
JSON nav-tabs were missing the ARIA roles Bootstrap's own tab pattern
expects (role="tab"/"tabpanel", aria-selected, aria-controls,
aria-labelledby) - screen readers had no way to tell these were tabs
or which pane went with which button. Added the standard Bootstrap 5
tab markup to both.
* admin: guard policy load/save flows against overlapping requests
1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy had no
protection against overlapping loads. Opening one bucket's (or
table's) policy dialog and then another's before the first GET
resolved let the late response write its document into the shared
textarea and mark the dialog "loaded" while it was now targeting the
second resource - a subsequent Save would then push the first
resource's policy onto the second. Add a per-load monotonic sequence
number (s3tablesBucketPolicyRequestSeq / s3tablesTablePolicyRequestSeq,
the same pattern already used for the classic bucket-policy load in
s3_buckets.templ); a response is only applied - textarea, loaded flag,
editor state - if its captured sequence still matches the latest one
issued.
Verified with a jsdom simulation: bucket A's policy load (artificially
slow) followed immediately by bucket B's (fast) previously left A's
policy in the textarea once A's late response landed; it now correctly
keeps B's.
2. s3_buckets.templ: the bucket-policy Save button lives outside the
(initially hidden) editor wrapper, so it stays clickable while a load
is still in flight - the existing policyRequestSeq guard only protects
the *load* from a stale response, not Save from firing before any
load for the current bucket has completed. Add bucketPolicyLoaded,
reset before each GET and set only once the matching response lands,
and check it at the top of the Save handler.
Verified with a jsdom simulation: clicking Save immediately after
opening the dialog, before a (deliberately never-resolving) GET
settles, now sends no PUT; a normal load-then-save sequence still
PUTs the loaded policy unchanged.
* admin: address further code review findings on the policy editor
1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy only
reset the JSON textarea when a new load starts; the structured editor
kept showing the previously loaded resource's statements (Editor tab
is the default active one) until the new fetch resolved. Call
loadPolicyTextareaIntoEditor() against the now-cleared textarea
immediately, so switching resources visibly resets the editor right
away instead of only once its own load completes. Verified with jsdom:
opening bucket A (loads fully) then bucket B (GET never resolves) no
longer leaves A's statements visible in B's editor.
2. s3tables.js: deleteS3TablesBucketPolicy/deleteS3TablesTablePolicy had
no loaded-state check, so a failed GET (which already blocks Save)
left Delete fully able to remove the resource's stored policy sight
unseen. Add the same s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded
guard Save already uses. Verified with jsdom: delete after a failed
load now sends no DELETE; delete after a successful load is unaffected.
3. s3_buckets.templ: the bucket-policy Editor/JSON nav-tabs were missing
the same ARIA roles already added to the S3 Tables policy tabs in an
earlier round (role="tab"/"tabpanel", aria-selected, aria-controls,
aria-labelledby) - this instance was out of scope for that review
comment but is the same gap. Bootstrap's own tab.js already manages
aria-selected on tab switch once the attribute exists, so no extra JS
was needed.
4. s3_buckets.templ: neither the bucket-policy Save nor Delete handler
guarded against a double-click, or against firing while the other was
still in flight - two overlapping PUT/DELETE requests for the same
bucket could land in either order. Add a shared
bucketPolicyMutationInFlight flag: set (and both buttons disabled)
before each fetch, cleared (and buttons re-enabled) on failure so the
user can retry, left set through the existing success hide-and-reload
path, and also reset when a new bucket's dialog opens so an abandoned
in-flight request from a closed dialog can't leave the buttons stuck
disabled. Verified with jsdom: double-clicking Save now sends exactly
one PUT, and a Delete click while that PUT is still pending sends no
DELETE.
* admin: scope bucket-policy mutation completions to the bucket that started them
1. The previous round's fix reset bucketPolicyMutationInFlight whenever a
new bucket's policy dialog opened, to avoid leaving Save/Delete stuck
disabled if the modal was closed mid-request. That traded one bug for
a worse one: if bucket A's PUT/DELETE was still in flight when the
user opened bucket B's dialog, the reset let B's Save/Delete fire
immediately, and A's completion handler - unaware anything had
changed - would still hide the (now B's) modal and reload the page
out from under whatever the user was doing with B, on success, or
alert a message with no bucket context, on failure.
Stop resetting on reopen, so a pending mutation for a previous bucket
keeps this bucket's Save/Delete blocked until it settles (matches the
"preventing overlapping mutations" the review comment describes).
Instead, capture policyEditorBucket as targetBucket right before each
fetch and compare it against policyEditorBucket again in the
completion handler: the in-flight flag is always released so the
buttons never get stuck, but the modal-hide/reload/alert only fire if
this bucket is still the one showing; a stale completion for an
abandoned bucket just logs to the console instead.
Verified with a jsdom simulation: opening bucket B while bucket A's
Save is still pending leaves B's Save button disabled and a click on
it a no-op; once A's PUT resolves, B's button re-enables but no
modal.hide()/reload() fires (previously both fired unconditionally).
2. bucketPolicyDeleteBtn had no bucketPolicyLoaded check, unlike Save -
a failed GET blocked Save but left Delete free to remove a policy the
client never actually saw (the same gap already fixed for the S3
Tables policy modals in an earlier round). Added the same guard,
ahead of the confirm() dialog. Verified with jsdom: Delete after a
failed load now sends no DELETE request.
* admin: fix spelling mistake
The storage rule's fsync decision was computed by the filer
(detectStorageOption -> rule.Fsync) and applied on the filer's own HTTP
write path, but was never carried onto the chunk uploads S3 issues: the
AssignVolumeResponse had no fsync field, so the s3api client could not
learn the decision, and the chunked upload URL was hardcoded without it.
Every S3 write to a path with fsync configured went to the volume server
as a non-fsync write.
Carry the decision through the assign response:
- filer.proto: AssignVolumeResponse gains bool fsync, filled from the
storage option the assign resolved.
- operation.AssignResult gains Fsync, so uploadChunk can append
?fsync=true to the volume server upload URL (single and replica
fan-out paths).
- The S3 PUT/UploadPart assignFunc, the S3 copy path, the admin file
browser upload, and the Iceberg worker assign functions all forward
the response field.
Adds TestUploadReaderInChunksAppendsFsyncWhenAssigned.
* erasure_coding: one home for the shard-count to volume-slots conversion
* ec: refund the cleared leftover shards' slots in the encode source health check
The Lance namespace gateway took the request-body location field, trimmed a
trailing slash, and passed it straight to the marker sink. That location feeds
TableDataDirFromMetadataLocation, which joins it under /buckets and collapses
any ../ segments, and writeMarker's CreateEntry then auto-creates every missing
parent. A caller could point the location at another tenant's bucket, or escape
/buckets entirely, and plant a fixed-name marker (recursively creating the
parents) or hide a victim's live table with .lance-deregistered.
Confine the declared location the way the Iceberg gateway already does: require
an s3:// URI whose bucket is the caller's own and whose path carries no
traversal segment, on both the declare and register handlers.
A client that reconnects before the old handler exits re-registers the
same client name, and addClient overwrites the map entry. The old
handler's deferred deleteClient then closed whatever channel the map
held under that name: the new, live stream's. Receiving from a closed
channel returns nil immediately and forever, so the new handler's send
loop degenerated into sending empty responses at wire speed, pinning a
core on each side until the client killed the connection.
deleteClient now closes the channel its own handler registered and
leaves the map entry alone unless it still points to that channel. This
also closes the previously orphaned old channel, whose drain goroutine
used to leak. The send loop treats a closed channel as an exit instead
of a message stream.
s3api: no filer failover after fn has consumed part of a response
withFilerClientFailover replays fn verbatim on the next filer, so a filer
that died mid-stream followed by a healthy peer returned success with the
callback's closure-captured accumulator holding the dead filer's prefix
twice; the per-attempt accumulator in listWithRetry could not close this,
because the replay happens inside a single attempt. Track delivery on the
connection handed to fn: once a unary reply or streamed message has reached
the callback, surface the transport error unwrapped instead of failing
over, and let callers replay from a clean slate. A filer that fails before
delivering anything fails over exactly as before.
* telemetry: sync the server module to go 1.26
The root module moved to go 1.26 but the telemetry server module, which
replaces seaweedfs with the repo root, stayed on 1.25.8, so go refuses
to build or test it until the directive catches up.
* telemetry: confirm a cluster after a week of reports, not two days
Two days of history still lets recurring CI and demo clusters into the
confirmed fleet: anything torn down and rebuilt across a UTC midnight
counts. Requiring seven distinct UTC days keeps the fleet charts and the
version/OS distributions to clusters that actually stay up; real
clusters qualify after their first week, and the fallback to all active
clusters while none is confirmed is unchanged.
* filer, volume server: return 416 when no requested range overlaps the content
* seaweed-volume: return 416 when no requested range overlaps the content
* server: check the range test error, use the request context, fix the no-overlap comment boundary
* master: never re-seed a raft cluster over committed state
-raftBootstrap deleted logs.dat, stable.dat and snapshots on every start and
then bootstrapped a fresh cluster. Since hashicorp raft only snapshots after
8192 log entries, the TopologyId lives in the log, not in a snapshot, so the
pre-wipe snapshot recovery found nothing and each restart minted a new cluster
identity. A master that came up while it could not reach its peers seeded a
rival cluster; when the two logs met, SetTopologyId's split-brain guard fatally
stopped every master holding the other id, and the master layer crash-looped
with no quorum.
Bootstrapping is genesis. Drop the wipe and the inline bootstrap. The first
master in -peers already mints a cluster once it has confirmed no peer has a
leader, so the flag has nothing left to do and is now ignored; keeping that one
master the sole bootstrap authority is what stops a partition from minting two
clusters, so the flag must not widen it either. A master with state rejoins its
peers, and one whose data dir was reset is admitted by the sitting leader
instead of forking again.
* test: cover -raftBootstrap restarts in the multi-master suite
Three masters start with -raftBootstrap, the way the helm chart renders it on
every master on every roll, and the cluster has to hold one TopologyId after
they all restart. /dir/status is proxied to the leader, so each master's own
view of the identity is read out of its log, which is where a fork shows up.
Before the fix the hashicorp case minted a new id on each restart.
* fix(mount): reply to LINK with the kernel node id, not the stored inode
Link() answered the kernel with out.NodeId = oldEntry.Attributes.Inode.
That attribute is a mount-runtime number and only entries created through a
mount carry one. An entry written by the S3 API, WebDAV or a direct filer
call persists inode 0, so the LINK reply named node id 0, which the kernel
rejects as invalid_nodeid and reports as EIO. The hard link itself had
already been written to the filer, which is why it looked correct again
after a mount restart.
The same stale number was also used as an inodeToPath key. AddPath(0, path)
filed the new link under inode 0, so a later Lookup on that name handed the
kernel node id 0 as well, and a LOOKUP reply carrying node id 0 means no
such entry.
in.Oldnodeid is the node id the kernel already holds for the source, and it
is the key inodeToPath is indexed by, so use it for the reply, for AddPath
and for the sibling sync.
Fixes#8404
* test(mount): cover the sibling sync in Link with a third hard link
The two existing cases never reach the body of syncHardLinkSiblings: with
two links the source alias and the name just created are both in skipPaths,
so the loop iterates over nothing and a change to that site goes unnoticed.
A third link leaves one name that no other part of Link() writes.
The new case drives three links off one source. It guards against covering
nothing (it fails if every path turns out to be a skipPath), checks that
every name of the file reports nlink 3, and then drives the sync with both
candidate keys to pin down which one it has to be: keyed by the source's
persisted Attributes.Inode, which is 0 for an entry written outside a mount,
GetAllPaths has no path to walk, while the kernel node id reaches the
sibling.
That second half is driven directly because Link() alone cannot tell the two
keys apart. The meta cache keeps one blob per hard link id (FilerStoreWrapper
setHardLink/maybeReadHardLink), so a read of any sibling returns the
attributes of the last write to any of them whether or not the sync ran.
* s3api: retry a transient failure when listing multipart uploads/parts
A blip on the way to the filer failed the whole ListMultipartUploads or
ListParts request. Both reported failure points sit inside one streaming
listing: the ListEntries call that opens the stream, and the stream.Recv
calls that drain it. Neither retried, so a single Unavailable answer from
a filer that was restarting turned into a 500 for the S3 client.
Replay the listing instead, bounded to three attempts with a 100ms
backoff that doubles. Only a transient failure is replayed. A not-found
answer stays authoritative so the empty-list branch still works, and
every other error still reaches the client on the first attempt.
This is scoped to (*S3ApiServer).list rather than added inside
DoSeaweedListWithSnapshot, which mount, the shell and the other object
listings share, and where a retry after a partial stream would
re-deliver entries the callback had already seen. Within one call to
list, a replay is safe: it collects into a fresh slice each time, so it
can neither duplicate nor drop entries.
That guarantee does not extend past this function. withFilerClientFailover
already re-runs its callback against the next filer on any non-NotFound
error without resetting the caller's accumulator, so on a multi-filer
gateway a mid-listing failover can itself produce a duplicated result
with err == nil, independent of this change and not fixed by it. Noted
in the PR rather than silently left for someone to rediscover.
Fixes#7221
References #7235
* s3api: move the listing retry inside list itself
---------
Co-authored-by: Junker der Provinz <jdp@braethoria.com>
* admin: honour persisted task configs when building the maintenance policy
buildPolicyFromTaskConfigs passed a literal nil to vacuum, erasure_coding
and balance LoadConfigFromPersistence. Those functions look for their
LoadXTaskPolicy() accessor via a type assertion, which a nil interface can
never satisfy, so every call fell through to NewDefaultConfig() and the
policy came back with the compiled-in defaults - Enabled: true among them.
A task disabled on disk was therefore still scheduled, and the only trace
was a glog.V(1) "Using default ... configuration" line.
Thread the real ConfigPersistence through instead. There are two copies of
this function: the one in weed/admin/dash builds config.Policy on the
normal admin startup path and can simply take cp as its receiver, and the
one in weed/admin/maintenance is the fallback used when the config carries
no policy yet, which now receives the store from NewMaintenanceManager.
weed/admin/dash already imports weed/admin/maintenance, so the maintenance
side has to keep the duck-typed interface{} parameter that the task
loaders already use rather than importing the concrete type back.
The store is only handed over when a data directory is configured: an
unconfigured one has nothing to read, and a typed nil pointer would pass
the loaders' type assertion and then panic on first use.
Fixes#10874
* admin: restore the maintenance scan cadence after an error backoff
scanLoop shortens its ticker to the error backoff delay after a failed
scan, but it decided whether to replace the ticker by comparing the
target interval against the configured scan interval instead of against
the interval the ticker was actually running at. Once the errors stopped,
getScanInterval returned the configured interval again, the comparison
came out false, and the ticker was left at the backoff delay - so a
single transient scan failure pinned the scanner to one scan per second
for the rest of the process lifetime. That is the ~1/second cadence in
issue #10874: 658 KB/s of "Cancelled N stale pending balance tasks
before re-detection" and 193k orphaned task files over two days.
Track the interval the ticker is running at and compare against that, so
both entering the backoff and returning to the normal cadence replace the
ticker.
While in here:
- defer ticker.Stop() bound the ticker that was current when the defer
was registered, so every replacement ticker leaked on return. Wrap it
in a closure.
- running was written by Start/Stop and read by all three background
loops without synchronisation. Guard it with the existing mutex, fold
the running check in triggerScanInternal into the lock it already
takes, and make Stop a no-op when not running so a second call cannot
close the stop channel twice.
Refs #10874
* admin: make the maintenance policy actually reach the task detectors
Loading the persisted task configs into the maintenance policy only
matters if something reads that policy, and nothing did.
MaintenanceIntegration pushes the policy into every registered detector
and scheduler through interface{ SetEnabled(bool) } and
interface{ SetMaxConcurrent(int) } type assertions. Every task registered
through base.RegisterTask is backed by base.GenericDetector and
base.GenericScheduler, and neither implemented either method, so all four
assertions failed silently for every task on every startup. The policy's
enabled flag reached nothing: ScanWithTaskDetectors gates on
detector.IsEnabled(), and the queue's policy lookups for max concurrent
and repeat interval are fallbacks that only fire when the scheduler
reports zero, which the generic scheduler never does.
Add the setters, delegating to the TaskConfig.SetEnabled the interface
already declares and to TaskDefinition.MaxConcurrent, which is what
GetMaxConcurrent returns.
Applying the policy required three more fixes, because with the
assertions working the policy could now do damage as well as good:
- IsTaskEnabled reports false for a task type the policy has no entry
for, so applying it unconditionally would have disabled every task the
policy does not list. Skip task types with no policy entry: no entry
means no opinion, not disabled.
- ec_balance was exactly such a task. It is registered like the other
three but had no entry in the policy builder and no accessor on
ConfigPersistence at all, so its configuration could never be
persisted. Add SaveEcBalanceTaskPolicy/LoadEcBalanceTaskPolicy, the
task_ec_balance.pb file, the SaveTaskPolicy dispatcher case, and the
policy entry.
- InitMaintenanceManager ran before loadTaskConfigurationsFromPersistence,
which replaces each task's whole config object, so the policy was
applied and then immediately thrown away. Swap the order. Both read the
same files, so the policy is now the last writer and stays
authoritative.
MaintenanceManager.UpdateConfig also updated the queue's and the
scanner's policy but not the integration's, so a policy changed at
runtime never reached the detectors. Add MaintenanceIntegration.SetPolicy
and call it.
While building the policy, stop hand-copying each task's fields and use
the task's own ToTaskPolicy(). The hand-written version was a second
definition of every task's policy and had already lost the erasure coding
preferred tags and replica placement and the balance IO rate limit. For
the same reason, the "nothing persisted yet" branches of
LoadVacuumTaskPolicy, LoadErasureCodingTaskPolicy and
LoadBalanceTaskPolicy now derive from each task's NewDefaultConfig()
instead of a third hand-written copy. Those copies had drifted, so with a
data directory but no config file on disk the effective defaults differed
from what the task and the admin UI schema both advertise:
vacuum scan interval 24h -> 2h
balance scan interval 6h -> 30m
balance imbalance 0.1 -> 0.2
erasure coding scan interval 168h -> 1h
erasure coding fullness 0.90 -> 0.95
erasure coding min volume 1024MB -> 30MB
Finally, weed/admin/dash and weed/admin/maintenance each carried a copy
of the policy builder and they had already diverged. Export the
maintenance one as BuildPolicyFromTaskConfigs and have dash call it.
Refs #10874
* worker: warn when a config store cannot supply a task's persisted config
LoadConfigFromPersistence logged a single glog.V(1) "Using default X
configuration" for every way of not loading anything, so the bug in
issue #10874 - a store handed in that the type assertion rejects, leaving
a task running on compiled-in defaults - looked exactly like the normal
"no data directory configured" case. The reporter had to read the source
to work out why their disabled task kept running, and asked for this
specifically.
Separate the cases. A non-nil store that does not provide the accessor is
always a wiring bug and is now logged at warning level, naming the type
and the missing method. A read error or a policy that will not apply is
also a warning. No persistence configured, and a store with nothing saved
yet, stay at V(1): those are normal.
Refs #10874
* admin: stop GetTaskPolicy panicking on a maintenance policy that is nil
GetTaskPolicy dereferenced its MaintenancePolicy argument to look at
TaskPolicies, so IsTaskEnabled, GetMaxConcurrent and GetRepeatInterval
all took the admin process down when handed a nil policy. A nil policy is
not a programming error here: MaintenanceConfig.Policy is unset until
something builds one, DefaultMaintenanceConfig returns a config with no
policy at all, and UpdateConfig installs whatever config it is given.
Found by calling IsTaskEnabled with the policy from a freshly defaulted
MaintenanceConfig.
Treat a nil policy as "no entry": no task enabled, the safe concurrency
default of 1, and a repeat interval of 0 so callers fall back to their
own default instead of reading DefaultRepeatIntervalSeconds off nil.
Also add the startup test this was found with. It walks the admin
server's startup sequence over a data directory that has balance saved as
disabled and checks the state that decides whether issue #10874 happens:
the balance detector reports disabled, vacuum stays enabled, and tasks
whose config was never saved keep their compiled-in default.
Refs #10874
* admin: document the synchronisation SetPolicy would need beyond startup
ConfigureTasksFromPolicy now really writes TaskDefinition.Config and
TaskDefinition.MaxConcurrent, which the scan loop reads through
detector.IsEnabled() with nothing synchronising the two. Every caller
runs during admin server startup today, before the scan loop exists, so
there is no live race - but the next caller has to add the locking, and
the same already applies to UpdateAllConfigs replacing the whole config
object. Write it down at the seam instead of leaving it to be
rediscovered.
Refs #10874
s3api: retry a transient failure when listing multipart uploads/parts
A blip on the way to the filer failed the whole ListMultipartUploads or
ListParts request. Both reported failure points sit inside one streaming
listing: the ListEntries call that opens the stream, and the stream.Recv
calls that drain it. Neither retried, so a single Unavailable answer from
a filer that was restarting turned into a 500 for the S3 client.
Replay the listing instead, bounded to three attempts with a 100ms
backoff that doubles. Only a transient failure is replayed. A not-found
answer stays authoritative so the empty-list branch still works, and
every other error still reaches the client on the first attempt.
This is scoped to (*S3ApiServer).list rather than added inside
DoSeaweedListWithSnapshot, which mount, the shell and the other object
listings share, and where a retry after a partial stream would
re-deliver entries the callback had already seen. Within one call to
list, a replay is safe: it collects into a fresh slice each time, so it
can neither duplicate nor drop entries.
That guarantee does not extend past this function. withFilerClientFailover
already re-runs its callback against the next filer on any non-NotFound
error without resetting the caller's accumulator, so on a multi-filer
gateway a mid-listing failover can itself produce a duplicated result
with err == nil, independent of this change and not fixed by it. Noted
in the PR rather than silently left for someone to rediscover.
Fixes#7221
References #7235
* admin: count plugin-runtime workers in worker metrics
The admin server keeps two worker registries: the legacy maintenance-worker
map, filled by workers registering over the worker gRPC stream, and the plugin
worker registry, filled by workers started as `weed worker`. Both the
SeaweedFS_admin_workers_connected / SeaweedFS_admin_worker_slots gauges and the
dashboard's Workers card read only the legacy map, so a cluster that runs the
admin and its workers as separate components reported 0 workers even while its
workers showed up on the plugin pages and ran scheduled jobs.
Aggregate both registries instead. The two are merged by worker ID: `weed mini`
starts both runtimes out of one working directory, so they share the persisted
worker ID and must not be counted twice. For such a worker the slot numbers
still come from the legacy registry, which keeps mini's existing readings.
Plugin workers report their slots in the heartbeat, so detection and execution
slots are summed from there; a worker that has connected but not yet sent a
heartbeat counts as connected with zero slots.
Fixes#10525
* admin: clamp negative worker-reported slot values in metrics merge
A plugin worker's self-reported heartbeat slot counts are untrusted
input; clamp them to 0 before summing so a stale or misbehaving
worker can't drive the aggregate gauge negative, matching the same
defensiveness already used in registry.go's own slot arithmetic.
* admin: add visual iam policy editor
Add a structured, tabbed editor (Editor / JSON) for creating and editing
IAM policies in the admin dashboard, alongside the existing raw-JSON
textarea:
- policies.templ: per-statement cards for Sid, Effect, Action, and
Resource, with unmanaged fields (Principal, NotPrincipal, NotResource,
Condition, or anything else) preserved verbatim in a per-statement
"advanced fields" JSON box so nothing is lost on round-trip. Switching
tabs commits and reparses in both directions. Restored the "Use Sample
Policy" button, now filling both the structured editor and the JSON
tab. The "Validate" button now calls the existing but previously
unused POST /api/object-store/policies/validate endpoint instead of
doing JS-only checks.
- Progressive Resource ARN autocomplete: suggests bucket names first,
then once "bucket/" is typed, suggests bucket/* plus the bucket's
direct subfolders, drilling down one path segment at a time as the
user types further "/" characters.
- New GET /api/files/list-folders endpoint (file_browser_handlers.go)
backing the folder autocomplete: wraps the existing file browser data
function and returns just the subdirectory names as JSON, scoped to
paths under /buckets.
- Action-name suggestions (datalist) for the Action field, sourced from
the existing s3_constants.S3_ACTION_* constants plus new
s3_constants.S3TABLES_ACTION_* constants (extracted from the s3tables
operation dispatch switch) so the suggestion list can't drift from the
strings the engines actually understand.
- policy_handlers.go: ValidatePolicy now accepts a statement with only
NotResource set (previously required Resource), matching
policy_engine.validateStatement and the fact the new editor makes such
statements reachable from the UI.
- Tests: ValidatePolicy behavior, route registration for the policy API
and the new list-folders endpoint, list-folders path scoping, and the
action-suggestion list's shape.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: fix XSS, cache poisoning, and cap overshoot in policy editor
Address code review findings on the IAM policy editor added in the
previous commit:
- policies.templ (displayPolicyDetails): escape every interpolated
policy value (Sid, Effect, Action, Resource, policy name, and the raw
JSON document) before assigning to innerHTML. Policy documents can
come from other admins or an import, so an unescaped field could
execute script when the "View" modal renders it.
- policies.templ (policyEditorStateToDoc): reject JSON arrays in a
statement's "advanced fields" box, not just invalid JSON. `typeof []
=== 'object'` was true, so a JSON array was assigned to the statement;
subsequent property assignments (Sid, Effect, ...) landed on the array
object but JSON.stringify of an array only serializes numeric indices,
silently dropping them.
- policies.templ (loadPolicyFolderNames): on a failed folder lookup,
remove the cache entry instead of permanently caching the empty
fallback, so a transient network/server error doesn't block retries
for the rest of the page's lifetime.
- file_browser_handlers.go (ListFolders): stop appending directory
names as soon as the running count reaches maxListFoldersEntries,
instead of only checking the cap after a full page is processed,
so the returned list never exceeds the configured cap.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001
to keep the diff scoped to this file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: stop policy editor from clobbering the active tab and dropping malformed advanced fields
Address two review findings on the IAM policy editor (Issue 3, stored-XSS
in displayPolicyDetails, was already fixed by the previous commit and is
unchanged here):
- createPolicy, updatePolicy, and validatePolicyDocument always committed
the structured editor's (possibly stale) state into the JSON textarea
before submitting, even when the user had just edited the JSON tab
directly. That silently discarded the user's JSON edits and
validated/saved the old structured-editor state instead, which could
leave broader permissions in force than intended.
Added commitPolicyActiveTab(which), which commits whichever tab is
currently visible into the other side instead of unconditionally
overwriting the JSON tab from the editor: if the JSON tab is active it
parses that JSON back into the structured editor (without touching the
textarea itself), otherwise it serializes the structured editor into
the textarea as before. All three call sites, plus the JSON-tab
"show.bs.tab" handler, now use this and abort with an alert if the
currently active tab's content can't be committed.
- policyEditorStateToDoc silently continued with an empty object when a
statement's "advanced fields" box held invalid JSON, so switching
tabs, validating, or saving would drop Principal/NotResource/Condition
from that statement without telling the user. It now throws (with the
statement number and parse error) on invalid or non-object JSON there,
and callers surface that via showAlert and abort instead of proceeding.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: keep unmanaged top-level policy fields across editor tab switches
policyDocToEditorState only carried Version and Statement into editor
state, so any other top-level key (e.g. Id) present in the JSON tab was
silently rewritten away as soon as the user switched to the Editor tab
and back. Capture those keys in state.otherFields and merge them back in
policyEditorStateToDoc before Version and Statement are written, so the
two tabs stay faithful to each other and the editor never rewrites text
the user typed.
Note this is editor fidelity only: the admin API's
policy_engine.PolicyDocument carries just Version and Statement, and
DocumentJSON is never populated, so such fields are still discarded by
the server once a policy is saved. Making them survive a save would
require a backend change, which is out of scope here.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: warn before a policy save discards unsupported top-level fields
The editor round-trips unmanaged top-level keys (e.g. Id) between the
Editor and JSON tabs, but the admin API's policy_engine.PolicyDocument
carries only Version and Statement, so the server drops them on save and
the user saw no indication.
Added confirmPolicyFieldDiscard(), called from createPolicy and
updatePolicy after the active tab is committed (so the field list is
accurate whichever tab is showing). It names the fields that will be
lost and lets the user confirm or cancel. Not wired into
validatePolicyDocument, which doesn't persist anything.
Chose the warning over the alternative of persisting these fields
through the backend: policy_engine.PolicyDocument is shared by the S3
bucket-policy engine and IAM evaluation, so extending it would change
the stored document shape for every policy in the codebase - far beyond
the scope of this editor.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: reject malformed Effect and Resource/NotResource conflicts in policy editor
Two review findings on the IAM policy editor:
- policyDocToEditorState defaulted any non-"Deny" Effect (missing,
misspelled, wrong case) to "Allow". A statement meant to be "Deny" with
a typo like "deny" would silently become a permissive "Allow" instead
of being rejected. It now throws on anything but an exact "Allow" or
"Deny", naming the offending statement and value.
commitPolicyTextareaToEditor catches this the same way it already
catches invalid JSON: alert the user and keep the JSON tab active
instead of switching to the Editor tab with wrong data.
- policyEditorStateToDoc could save a statement with both Resource (from
the structured field) and NotResource (surviving in the "advanced
fields" extras from before the user switched to using Resource) set at
once - a contradictory combination neither the admin's ValidatePolicy
handler nor policy_engine's evaluator rejected. When the structured
Resource field is non-empty it now deletes any leftover NotResource
from extras, consistent with the file's existing rule that structured
fields take precedence over extras. Mirrored the existing
Principal/NotPrincipal exclusivity check in
weed/admin/handlers/policy_handlers.go's ValidatePolicy to reject the
same combination server-side, since create/update perform no
validation at all. Deliberately left policy_engine.validateStatement
(used by the S3 bucket-policy PUT handler for every bucket policy in
the product) unchanged - extending that shared validator is a larger,
separate change outside this admin-editor fix's scope.
Added a handler test for the new Resource+NotResource rejection.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: add NotResource support to the visual policy editor
Since Resource and NotResource are mutually exclusive (enforced by a
previous fix), NotResource could previously only be set through the raw
JSON in a statement's "advanced fields" box. Promote it to a first-class
mode of the structured editor:
- The static "Resources" label is now a Resource/NotResource dropdown;
the same list of values underneath is reused for either key depending
on the selected mode, with a short form-text explaining the semantics.
- NotResource is added to POLICY_STATEMENT_KNOWN_KEYS, since it's now a
managed field like Resource rather than something that falls through
to extras.
- policyDocToEditorState derives resourceMode from which key is present
on load, and throws (same handling as the existing malformed-Effect
case: alert, keep the JSON tab active) if a hand-edited document has
both Resource and NotResource on one statement, since that can't be
represented by the dropdown.
- policyEditorStateToDoc writes only the key matching the selected mode,
replacing the previous one-directional "delete NotResource whenever
Resource is set" fix with mode-driven logic that also deletes Resource
when NotResource is selected.
- displayPolicyDetails (the read-only View modal) now shows the actual
NotResource values with a distinct label instead of a static
"(NotResource used instead)" placeholder.
No backend changes: the server-side "cannot specify both" check added
previously in policy_handlers.go's ValidatePolicy already covers this.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: reject non-object policy documents; catch Principal/NotPrincipal conflicts server-side
Two review findings:
- policyDocToEditorState treated a top-level JSON value that wasn't an
object (null, or a bare string/number/boolean) as an empty statement
list instead of failing explicitly. If the user typed e.g. "hello" or
42 in the JSON tab and switched to the Editor tab, their input was
silently discarded and replaced with an empty policy - the same class
of "guess instead of reject" bug fixed for malformed Effect and
Resource/NotResource conflicts previously. Added an explicit check
that throws for null/scalar input, while leaving array and object
document shapes accepted exactly as before.
- weed/admin/handlers/policy_handlers.go's ValidatePolicy checked the
Resource/NotResource conflict by non-empty length
(len(...Strings()) > 0), which misses a statement where Resource is
explicitly present but an empty list (e.g. "Resource": []) alongside a
non-empty NotResource. Switched that check to field presence (!= nil),
matching how policy_engine's own validateStatement already treats
Principal/NotPrincipal exclusivity. Also added the equivalent
Principal/NotPrincipal presence check to this handler, which had none
before - the advanced-fields box in the visual editor lets a user set
both today, and nothing server-side caught it. The existing
non-empty "Resource or NotResource is required" check is left as a
length check, since an empty array shouldn't count as "provided".
Added test cases for both conflict checks in policy_handlers_test.go.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: add Principal/NotPrincipal support to the visual policy editor (v1, AWS-only)
Adds a first, deliberately narrow structured editor for a statement's
Principal/NotPrincipal, left out when NotResource support was added:
- A Principal/NotPrincipal mode dropdown mirrors the existing
Resource/NotResource one (same mutual-exclusivity handling: the two
fields can't be set at once, and switching modes reuses the same
value list).
- A simple repeatable text-value list feeds a single {"AWS": [...]}
object on save - always the AWS type, never the "bare" (untyped)
SeaweedFS-extension shape. Per policy_engine's allowedPrincipalKeys,
Service/Federated/CanonicalUser also parse successfully, but nothing
in the S3 bucket-policy evaluation path ever sets a real caller's
principal to a service name, an OIDC provider ARN, or a canonical
user ID, so only AWS is functionally meaningful today - out of scope
for this v1.
- On load, only the exact {"AWS": ...} single-key shape is unwrapped
into the structured field and removed from "extras". Anything else
(bare string/array, a different single type key, or several type
keys at once) is left untouched in "extras" exactly as before, with a
visible warning under the dropdown so the user knows a
Principal/NotPrincipal exists but isn't shown there. Saving with the
structured field left empty never touches whatever's already in
extras, so a preserved complex form isn't silently dropped just
because the user didn't touch this field.
- The read-only View modal now displays Principal/NotPrincipal for any
shape (via a small generic summarizer), not just the AWS-simple one.
- Generalized the action/resource field-to-state-key mapping (used by
commitPolicyEditorForm and the add/remove-item click handler) into a
shared lookup table instead of stacking another ternary, now that a
third field (principal) exists.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: support the bare "*" wildcard Principal in the visual editor
"Principal": "*" (and NotPrincipal: "*") is the standard AWS shorthand
for "everyone" and is common in real bucket policies, but the v1
Principal/NotPrincipal editor only recognized the {"AWS": ...} object
form, leaving a bare "*" statement's principal hidden in Advanced
fields.
parseSimpleAwsPrincipal now also accepts the bare string "*" as a
simple, structurally-editable value. On save, a principal value list
containing exactly ["*"] is written back as the bare "*" string
(matching the common convention) rather than wrapped as {"AWS": "*"};
anything else still wraps under AWS as before. Updated the field's
form-text hint accordingly.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: add Principal field autocomplete backed by users + IAM roles
Adds a datalist-backed autocomplete for the policy editor's Principal/
NotPrincipal text fields, sourced from a new API listing existing
identities:
- weed/admin/dash/principal_suggestions.go: AdminServer.GetPrincipalSuggestions
combines S3 user ARNs (via the existing GetObjectStoreUsers +
iam.UserArn) with IAM role ARNs (via integration.NewFilerRoleStore /
ListRoles, reusing the exact same construction already used in
iam_manager.go - no new dependency risk introduced). Role ARNs are
reconstructed from the role name using SeaweedFS's default
arn:aws:iam::role/<name> convention rather than fetching each role's
stored definition, since this only backs a suggestion list. Role
listing failures are logged and swallowed rather than failing the
whole request - an incomplete suggestion list is fine, blocking
policy editing over it is not. Service accounts are deliberately not
listed separately: a service account's ARN is identical to its parent
user's, already covered by the user list.
- weed/admin/handlers/policy_handlers.go: GetPrincipalSuggestions handler
exposing this as {"principals": [...]}.
- Route registered at the API root (GET /api/principals) rather than
under policyApi's "/object-store/policies" prefix, since that
subrouter's existing "/{name}" GET route would shadow any
single-segment GET route registered after it (the same class of
gotcha previously seen with "/validate").
- weed/admin/view/app/policies.templ: a shared, lazily-fetched-once
policyPrincipalSuggestions datalist (flat list - unlike the
progressive per-folder Resource ARN autocomplete, users/roles aren't
hierarchical), wired into policyListRowHtml for field:"principal" and
populated on input/focus, with "*" always offered first.
Added tests for the new ARN-construction helper and route registration.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: fix fieldset/legend styling in the structured policy editor
Bootstrap's form reset stretches <legend> to the fieldset's full width
(float: left; width: 100%), which loses the native "notch in the
border" look and makes each section's label bar as wide as the card.
Add two scoped classes: .policy-stmt-fieldset (border, rounded
corners, spacing between sections) and .policy-stmt-legend (undoes the
float/width so the legend hugs its content, with a little padding).
Applied to the three per-statement sections (Actions,
Resource/NotResource, Principal/NotPrincipal), replacing the ad hoc
"border rounded" utility classes that were doubling up with the
fieldset's own border. Also gave the "Advanced fields" <details> a
small top margin to match the new spacing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: suggest bucket/* alongside the bucket itself in Resource autocomplete
At the bucket-name stage of the Resource field's progressive
autocomplete, only "arn:aws:s3:::bucket" was offered. Add
"arn:aws:s3:::bucket/*" right alongside it, since granting access to
everything in a bucket is the more common case and previously required
typing a "/" first to reach the folder-level "*" suggestion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: keep an unparseable policy in the JSON tab instead of wiping it
editPolicy() built the structured state inside the fetch .then, so a
policy the editor cannot model threw into the sibling .catch, which
alerted and called hide(). Showing the alert at that moment left the
modal on screen with an empty editor and the document only in the JSON
tab, and Save Changes then serialized the empty state over the policy.
Reachable two ways, since neither create path rejects these: the admin
API never validates on create, so "Effect":"allow" is stored as typed,
and policy_engine.validateStatement lets Resource and NotResource sit
in the same statement.
Hand the document to the JSON tab instead, which is what that tab is
for, and mark the state so nothing serializes the placeholder over it.
* admin: validate a policy document before saving it
Validation was wired only to the Validate button, so nothing stopped a
document the server's own validator rejects from being stored. With the
structured editor supplying the boilerplate and required dropped from
the textarea, opening the modal, typing a name and clicking Create
Policy was enough to save a statement-less policy.
Share validatePolicyJSON with the two save paths and abort on failure.
* admin: bound the folder autocomplete listing
maxListFoldersEntries caps the folders collected, but nothing capped the
entries paged through to find them, so a bucket holding only flat object
keys - no subfolders to count - was walked to the end, 200 entries per
round trip, behind one keystroke. Measured against an in-process filer:
6 entries 0.5ms, 3k entries 7.7ms, 30k entries 53ms, all of it linear in
the directory rather than in the answer.
Cap the scan as well, and let GetFileBrowser take a prefix so the segment
the user is still typing is filtered by the filer instead of by paging.
The same 30k directory now answers in 0.6ms once a prefix is typed.
* admin: clean the path before scoping list-folders to /buckets
util.CleanWindowsPath only rewrites backslashes, so "/buckets/../etc"
walked straight past the prefix check the endpoint relies on for its
scope. Nothing leaked - filer paths are literal keys, so the traversal
resolved to nothing - but the check reads as a boundary and wasn't one,
and the test asserting it didn't cover the one input that would try.
validateAndCleanFilePath in the same file already does this.
* admin: stringify policy values before escaping them
escapeHtml calls text.replace directly, and the Sid, the per-item action
and resource inputs, and the View modal's Resource/NotResource all pass
values straight out of JSON.parse. A policy carrying "Sid": 5 or
"Action": [1] threw "text.replace is not a function" and took the render
with it. escapedJoin already coerced; use it everywhere and coerce the
editor state at the point it's built.
* admin: only show the NotResource hint in NotResource mode
The hint rendered unconditionally, so it sat under a selector reading
"Resource" telling the user the statement applies to everything except
what they'd listed. Redraw the card when the selector changes so it
follows the mode.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* install.sh: install the Rust maintenance worker
The release publishes weed-worker but the installer only knew weed and the
Rust volume server, so the one binary that cannot be built without a Rust
toolchain was the one you had to download by hand.
--component all skips it on a platform it has no build for rather than failing
an install that already put two binaries in place; asking for it by name there
still says so.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* install.sh: clean each component's temp directory as it finishes
The EXIT trap is per-process, so installing more than one component left every
extraction but the last behind. Cleaning at the end of the function keeps the
trap for the paths that exit early.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* worker: compile plugin.proto with the protoc that ships with the build
seaweed-volume already does this: protoc-bin-vendored carries the binary, so
the build needs no package manager and every build sees the same version. An
explicit PROTOC still wins, which is what lets the lance crates - whose own
build scripts read the same variable - share it.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: point the worker builds at the vendored protoc
The jobs installed protobuf-compiler for lance's build scripts. They read
PROTOC, so pointing it at the binary protoc-bin-vendored already puts in the
registry serves them without a system package - one less apt call on the way
to a release, and the same protoc a developer's build uses.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docs: say what the worker build needs from protoc
The lance crates' build scripts are the ones that need it, not ours, and they
take the same vendored binary.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* worker: name the binary weed-worker
It is the Rust side of `weed worker`, the way weed-volume is the Rust side of
`weed volume`, and lance is the first family of jobs it carries rather than
the only one it ever will. The crate keeps its own name: when a second family
arrives the bin target moves to a crate of its own, under this name.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docker: ship the Rust maintenance worker in the image
Lance table buckets need a worker that can read the format, and until now the
only way to get one was a Rust toolchain and a cargo build. It now sits at
/usr/bin/weed-worker beside the Rust volume server, reached as
`docker run chrislusf/seaweedfs worker-rust --admin host:23646` — the verb
mirrors volume-rust, so plain `worker` still runs the Go one.
Taken pre-built or not at all: the lance jobs pull in arrow and datafusion, far
too large a tree to compile inside the image build, so an architecture CI did
not build for gets the empty placeholder the entrypoint refuses to exec, the
way the Rust volume server already does.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: build the Rust worker for the container images
The same native cross-compile the volume server uses, so the release, latest
and dev images all carry it on amd64 and arm64. The artifact holds both
binaries now, so it is named for that rather than for the volume server.
Only the release directory each job builds is cached: with a debug profile
beside it the worker's target/ reaches 24GB, against a 10GB cache budget.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: publish Rust worker binaries with the release
Linux amd64 and arm64 only: the worker runs beside the cluster it maintains,
and its dependency tree makes every extra target an expensive build.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: build and test the Rust workers on change
Nothing built seaweed-worker in CI, so the release and the container images
would have been the first place a break showed up. Tests run in release too,
rather than compiling lance, arrow and datafusion again in another profile.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docs: say how to get a released worker
Neither the image nor the release tarballs were mentioned; a toolchain and a
cargo build read as the only way in.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: install protoc for the Rust worker builds
lance's crates compile their own protos, and unlike seaweed-volume they do not
vendor a protoc to do it with, so every job that builds the worker failed at
lance-encoding's build script.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: do not persist credentials in the worker release checkout
The upload step is handed a token explicitly; a cargo build script should not
find another one sitting in the checkout's git config.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docker: keep the worker's argument boundaries
Unquoted $@ splits on whitespace and expands globs, so an argument carrying
either arrived as something else.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* helm: install chart CI against an image tag that exists
The release bumps appVersion on master well before the container build
publishes that tag, and the chart CI runs on the bump commit, so every
release turns it red with ImagePullBackOff. Resolve the tag first and
fall back to latest while the new one is still building.
* helm: run the chart CI when the workflow itself changes
* helm: bound the registry lookup in the chart CI
An unbounded curl can hold the job, and the log did not say why the tag
was rejected. Cap it and print the status.
2026-08-22 09:42:28 -07:00
1323 changed files with 127184 additions and 34706 deletions
org.opencontainers.image.description=SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast!
org.opencontainers.image.description=SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast!
foreach ($f in 'C:\seaweed-mount.log','C:\seaweed-mount.err.log','C:\seaweed-remount.log','C:\seaweed-remount.err.log','C:\seaweed-remount2.log','C:\seaweed-remount2.err.log','C:\seaweed-dirmount.log','C:\seaweed-dirmount.err.log','C:\seaweed-mini.log','C:\seaweed-mini.err.log') {
# The suite expects RGW's 400 InvalidPart for a partNumber past the last
# part; AWS answers 416 InvalidPartNumber, which is what we return.
sed -i "/# request PartNumber out of range/,+4{s/assert status == 400/assert status == 416/; s/assert error_code == 'InvalidPart'/assert error_code == 'InvalidPartNumber'/}" s3tests/functional/test_s3.py
# The suite expects RGW's 400 InvalidPart for a partNumber past the last
# part; AWS answers 416 InvalidPartNumber, which is what we return.
sed -i "/# request PartNumber out of range/,+4{s/assert status == 400/assert status == 416/; s/assert error_code == 'InvalidPart'/assert error_code == 'InvalidPartNumber'/}" s3tests/functional/test_s3.py
One `weed` binary serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Both read and write operations have O(1) complexity and can run at the full speed supported by the underlying hardware.
- [Download Binaries for different platforms](https://github.com/seaweedfs/seaweedfs/releases/latest)
- [SeaweedFS on Slack](https://join.slack.com/t/seaweedfs/shared_invite/enQtMzI4MTMwMjU2MzA3LTEyYzZmZWYzOGQ3MDJlZWMzYmI0OTE4OTJiZjJjODBmMzUxNmYwODg0YjY3MTNlMjBmZDQ1NzQ5NDJhZWI2ZmY)
- [SeaweedFS on Twitter](https://twitter.com/SeaweedFS)
- [SeaweedFS on Telegram](https://t.me/Seaweedfs)
- [SeaweedFS on Reddit](https://www.reddit.com/r/SeaweedFS/)
- [SeaweedFS White Paper](https://github.com/seaweedfs/seaweedfs/wiki/SeaweedFS_Architecture.pdf) and introduction slides: [2025.5](https://docs.google.com/presentation/d/1tdkp45J01oRV68dIm4yoTXKJDof-EhainlA0LMXexQE/edit?usp=sharing), [2021.5](https://docs.google.com/presentation/d/1DcxKWlINc-HNCjhYeERkpGXXm6nTCES8mi2W5G0Z4Ts/edit?usp=sharing), [2019.3](https://www.slideshare.net/chrislusf/seaweedfs-introduction)
Table of Contents
=================
* [Quick Start](#quick-start)
* [Quick Start with weed mini](#quick-start-with-weed-mini)
* [Quick Start for S3 API on Docker](#quick-start-for-s3-api-on-docker)
* [Introduction](#introduction)
* [Features](#features)
* [Additional Features](#additional-features)
* [Filer Features](#filer-features)
* [Example: Using Seaweed Blob Store](#example-using-seaweed-blob-store)
* [Architecture](#object-store-architecture)
* [Compared to Other File Systems](#compared-to-other-file-systems)
* [One command](#one-command)
* [Docker](#docker)
* [Docker Compose](#docker-compose)
* [Kubernetes with Helm](#kubernetes-with-helm)
* [Build from source](#build-from-source)
* [Scale out](#scale-out)
* [Why SeaweedFS](#why-seaweedfs)
* [Fast](#fast)
* [Scalable](#scalable)
* [The most complete S3 API](#the-most-complete-s3-api)
* [A data warehouse with S3 Tables](#a-data-warehouse-with-s3-tables)
* [A fast cache for cloud storage](#a-fast-cache-for-cloud-storage)
* [Active-active replication and more](#active-active-replication-and-more)
* [Architecture](#architecture)
* [Compared to Other Systems](#compared-to-other-systems)
* [Compared to HDFS](#compared-to-hdfs)
* [Compared to GlusterFS, Ceph](#compared-to-glusterfs-ceph)
* [Compared to GlusterFS](#compared-to-glusterfs)
* [Compared to MooseFS](#compared-to-moosefs)
* [Compared to Ceph](#compared-to-ceph)
* [Compared to MinIO, RustFS](#compared-to-minio-rustfs)
* [Dev Plan](#dev-plan)
* [Installation Guide](#installation-guide)
* [Disk Related Topics](#disk-related-topics)
* [Benchmark](#benchmark)
* [Enterprise](#enterprise)
* [License](#license)
* [Sponsors](#sponsors)
# Quick Start #
## One command ##
## Quick Start with weed mini ##
Download the latest binary from the [releases](https://github.com/seaweedfs/seaweedfs/releases/latest) page and unzip the single `weed` (or `weed.exe`) file, or let the install script put it in `/usr/local/bin`:
Download the latest binary from https://github.com/seaweedfs/seaweedfs/releases and unzip the single `weed` (or `weed.exe`) file, or run `go install github.com/seaweedfs/seaweedfs/weed@latest`. Then start a ready-to-use S3 object store with credentials and a pre-created bucket in one command:
That's it — the S3 endpoint is at http://localhost:8333, `my-bucket` already exists, and `admin`/`secret` are valid credentials. `S3_BUCKET` accepts a comma-separated list (e.g. `raw,processed`); use `S3_TABLE_BUCKET` for S3 Tables buckets, each `name` or `name:FORMAT` where the format is `ICEBERG` (the default) or `LANCE`. Drop any of the env vars to skip that piece (no AWS keys → S3 runs in unauthenticated "Allow All" mode for development).
That's it. The S3 endpoint is at http://localhost:8333, `my-bucket` exists, and `admin`/`secret` are valid credentials:
The same process also runs the master, a volume server, the filer, WebDAV, the Iceberg REST catalog, and the Admin UI. Add `S3_TABLE_BUCKET=warehouse` to also create an Iceberg table bucket, or `warehouse:LANCE` for a Lance one. Drop the AWS keys to run without authentication for development.
> macOS: if the binary is quarantined, run `xattr -d com.apple.quarantine ./weed` first.
Perfect for development, testing, learning SeaweedFS, and single-node deployments. To scale out, add more volume servers by running `weed volume -dir="/some/data/dir2" -master="<master_host>:9333" -port=8081` locally, on another machine, or on thousands of machines.
`weed mini` is auto-tuned for one node and is fine for single-node production, such as an S3 gateway that issues presigned URLs. See [Quick Start with weed mini][WeedMini].
## Quick Start for S3 API on Docker ##
## Docker ##
```bash
docker run -p 8333:8333 \
docker run -p 8333:8333 -v weed-data:/data \
-e AWS_ACCESS_KEY_ID=admin \
-e AWS_SECRET_ACCESS_KEY=secret \
-e S3_BUCKET=my-bucket \
chrislusf/seaweedfs
```
Same behavior as the `weed mini` command above — the S3 endpoint is at http://localhost:8333 with `my-bucket` pre-created. Drop the env vars to run anonymously for development.
Same behavior as the `weed mini` command above.
# Introduction #
## Docker Compose ##
SeaweedFS is a simple and highly scalable distributed file system. There are two objectives:
To run master, volume server, filer, S3, and WebDAV as separate services:
docker compose -f seaweedfs-compose.yml -p seaweedfs up
```
SeaweedFS started as a blob store to handle small files efficiently.
Instead of managing all file metadata in a central master,
the central master only manages volumes on volume servers,
and these volume servers manage files and their metadata.
This relieves concurrency pressure from the central master and spreads file metadata into volume servers,
allowing faster file access (O(1), usually just one disk read operation).
[Docker Compose for S3][DockerComposeS3] adds credentials, and the [docker/compose](docker/compose) folder has variants for replication, mounts, message queues, and more.
There is only 40 bytes of disk storage overhead for each file's metadata.
It is so simple with O(1) disk reads that you are welcome to challenge the performance with your actual use cases.
## Kubernetes with Helm ##
SeaweedFS started by implementing [Facebook's Haystack design paper](http://www.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf).
Also, SeaweedFS implements erasure coding with ideas from
[f4: Facebook’s Warm BLOB Storage System](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-muralidhar.pdf), and has a lot of similarities with [Facebook’s Tectonic Filesystem](https://www.usenix.org/system/files/fast21-pan.pdf) and [Google's Colossus File System](https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system)
A production-shaped `values.yaml` for a three-node cluster: two copies of every write, three masters, and an S3 endpoint with credentials and a bucket.
SeaweedFS can transparently integrate with the cloud.
With hot data on local cluster, and warm data on the cloud with O(1) access time,
SeaweedFS can achieve both fast local access time and elastic cloud storage capacity.
What's more, the cloud storage access API cost is minimized.
Faster and cheaper than direct cloud storage!
```yaml
global:
seaweedfs:
enableReplication:true
replicationPlacement:"001"# one extra copy on another server; "002" for two
SeaweedFS also ships a built-in **Iceberg REST Catalog**, turning the same cluster into a self-contained lakehouse.
Spark, Trino, Dremio, DuckDB, and RisingWave can query Iceberg tables directly — no Hive Metastore, Glue, or
external catalog service required. Storage and table metadata live in one system, simplifying on-prem and
small-team analytics stacks.
master:
replicas:3
data:
type:persistentVolumeClaim # the cluster's default storage class; add storageClass to pick one
size:1Gi
volume:
replicas:3# at least 1 + the sum of the replication digits
dataDirs:
- name:data
type:persistentVolumeClaim
size:500Gi
maxVolumes:0# size the volume count from the disk
filer:
replicas:2
data:
type:persistentVolumeClaim
size:20Gi
s3:
enabled:true
replicas:2
enableAuth:true
credentials:
admin:
accessKey:admin
secretKey:change-me
createBuckets:
- name:app-storage
```
The S3 endpoint is the `seaweedfs-s3` service on port 8333. [Helm Chart Recipes][HelmRecipes] has values for a development cluster, a lakehouse with the Iceberg catalog exposed, filer metadata on PostgreSQL, and node-local disks. The [SeaweedFS Operator][Operator] and the [CSI driver][SeaweedFsCsiDriver] are the other Kubernetes paths.
`weed` lands in `$GOPATH/bin`. [Getting Started][GettingStarted] covers running master, volume, filer, and S3 as separate processes.
## Scale out ##
Capacity is a volume server. Start one on any machine with disk and point it at the master:
```bash
weed volume -dir=/data -master=<master_host>:9333
```
Nothing rebalances until you ask it to. Throughput is a filer or S3 gateway; they are stateless, so run as many as you need behind a load balancer. [Production Setup][ProductionSetup] walks through a multi-node cluster.
[Back to TOC](#table-of-contents)
# Features #
## Additional Blob Store Features ##
* Support different replication levels, with rack and data center aware.
* Automatic master servers failover - no single point of failure (SPOF).
*Automatic compression depending on file MIME type.
*Automatic compaction to reclaim disk space after deletion or update.
*Flexible Capacity Expansion: Any server with some disk space can add to the total storage space.
*Adding/Removing servers does **not** cause any data re-balancing unless triggered by admin commands.
* Optional picture resizing.
* Support ETag, Accept-Range, Last-Modified, etc.
* Support in-memory/leveldb/readonly mode tuning for memory/performance balance.
* Support rebalancing the writable and readonly volumes.
* [Customizable Multiple Storage Tiers][TieredStorage]: Customizable storage disk types to balance performance and cost.
*[Transparent cloud integration][CloudTier]: unlimited capacity via tiered cloud storage for warm data.
*[Erasure Coding for warm storage][ErasureCoding] Rack-Aware 10.4 erasure coding reduces storage cost and increases availability. Enterprise version can customize EC ratio.
# Why SeaweedFS #
## Fast ##
*One disk read per blob. A small file is one blob; a large file is split into chunks of a few MB, each its own blob. A volume server keeps a 16-byte index entry per blob in memory and reads it in a single seek, also for erasure-coded data.
*The master is not in the read path. Clients cache the volume-to-server mapping and talk to volume servers directly.
*40 bytes of metadata per file on disk. Small files are packed into append-only volume files, so there is no per-file inode, no per-file metadata file, no fragmentation, and writes are SSD friendly.
*Hot data is replicated; [erasure coding][ErasureCoding] is applied to warm data in the background, so writes never pay the encoding cost.
*The [Rust volume server][RustVolume] is a drop-in for higher throughput and lower tail latency on the same on-disk format.
On one laptop, [`weed benchmark`][Benchmarks] writes 1KB files at 15,700 per second and reads them back at 47,000 per second, and a mixed S3 [warp][S3Benchmark] run totals 3.2 GiB/s. Numbers are in the [Benchmark](#benchmark) section; throughput grows with volume servers and gateways.
## Scalable ##
*The master tracks volumes, not files. A cluster with billions of files has a few thousand volumes, so the master stays small. One master is enough for most clusters; run three for [Raft failover][FailoverMaster].
*Adding a server adds capacity with no data reshuffle. Balancing, vacuum, erasure coding, and repair run on demand from [`weed shell`][WeedShell] or the [maintenance worker][Worker].
* Filer and S3 gateways are stateless and scale linearly. Directory metadata lives in a [store you already run][FilerStores]: LevelDB, RocksDB, SQLite, MySQL, PostgreSQL, Cassandra, HBase, MongoDB, Redis, Elasticsearch, etcd, TiKV, FoundationDB, YDB, ArangoDB, Tarantool, and MySQL or PostgreSQL compatible databases such as TiDB, CockroachDB, and MemSQL.
* Rack and data center aware [replication][Replication], [tiered storage][TieredStorage] across disk types, and [transparent cloud tiering][CloudTier] for unlimited capacity.
* Files from a byte to [tens of TB][SuperLargeFiles]. Volumes up to 8TB with the large-disk build.
## The most complete S3 API ##
The S3 gateway implements the object, bucket, S3 Tables, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged.
| API | Operations |
| --- | --- |
| S3 bucket and object | 73 |
| S3 Tables | 36 |
| IAM | 39 |
| STS | 5 |
* [Versioning][Versioning], [Object Lock][ObjectLock] with retention and legal hold, [lifecycle][Lifecycle] rules, tagging, [CORS][CORS], [conditional reads and writes][ConditionalOps], checksums, presigned URLs, browser POST uploads, multipart uploads, and an atomic [RenameObject][RenameObject].
* [Bucket policies][BucketPolicies] with [conditions][PolicyConditions] and [variables][PolicyVariables]; IAM users, groups, and policies; STS with [OIDC][OIDC], LDAP, and [Kubernetes service accounts][K8sSA].
* [SSE-S3, SSE-KMS, and SSE-C][SSE] server-side encryption, with OpenBao and Vault, AWS KMS, Azure Key Vault, and GCP KMS as key providers.
* [Audit log][AuditLog], [bucket quota][BucketQuota], and [rate limiting][RateLimiting].
* Each bucket is its own collection, so deleting a bucket is instant.
The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs vs MinIO][S3vsMinio] compares. The S3 compatibility suite and the SDK, IAM, SSE, policy, and Spark integration tests run in CI on every change.
## A data warehouse with S3 Tables ##
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials.
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
* IAM at the bucket, namespace, and table level with standard bucket policies, see [S3 Tables Security][S3TablesSecurity].
* A [Hadoop compatible file system][Hadoop] for Spark, Flink, and HBase.
`S3_TABLE_BUCKET=warehouse ./weed mini -dir=./data` brings the whole stack up on a laptop.
## A fast cache for cloud storage ##
[Cloud Drive][CloudDrive] mounts a bucket from S3, Google Cloud Storage, Azure, Backblaze B2, Wasabi, Storj, or any S3-compatible store into SeaweedFS and serves it at local speed:
* Metadata is pulled once, so listing, stat, and directory walks cost no cloud API calls.
* File content is downloaded once, on first read or [warmed][CacheRemote] by folder, name pattern, size, or age, and cached with the capacity of the whole cluster: cache everything, no churn.
* Local writes complete at local latency and are written back to the cloud asynchronously in the cloud's native layout, so other tools keep reading the bucket directly.
* Uncache by the same rules to free local disk while keeping the metadata.
[Cloud Tier][CloudTier] goes the other direction, moving whole warm volumes to cloud storage while keeping one-read access, and the [Gateway to Remote Object Storage][GatewayToRemoteObjectStore] mirrors every bucket to a remote store. Faster and cheaper than reading the cloud directly.
## Active-active replication and more ##
* [Active-active or active-passive replication][ActiveActiveAsyncReplication] between clusters, continuous and resumable, for the whole tree or chosen folders, across data centers.
* [Filer store replication][FilerStoreReplication] for metadata HA, [async backup][AsyncBackup] to cloud storage, [metadata backup][MetaBackup], and [change data capture][CDC] with [webhooks][Webhook] on every metadata event.
* The same data as a [FUSE mount][Mount] on Linux, macOS, and [Windows][MountWindows], over [WebDAV][WebDAV], [SFTP][SFTP], HDFS, HTTP, and [TUS resumable uploads][TUS]; on Kubernetes through the [CSI driver][SeaweedFsCsiDriver] and [Operator][Operator].
* [AES256-GCM encryption at rest][FilerDataEncryption], TLS and mTLS between components, JWT-signed volume access, and [FIPS][FIPS] builds.
* [Admin UI][AdminUI], Prometheus [metrics][Metrics], [TTL][VolumeServerTTL] per file or volume, automatic compression and compaction, and [seaweed-up][SeaweedUp] for bare-metal clusters.
[Back to TOC](#table-of-contents)
## Filer Features ##
* [Filer server][Filer] provides "normal" directories and files via HTTP.
* [File TTL][FilerTTL] automatically expires file metadata and actual file data.
* [Mount filer][Mount] reads and writes files directly as a local directory via FUSE.
* [Filer Store Replication][FilerStoreReplication] enables HA for filer meta data stores.
* [Amazon S3 compatible API][AmazonS3API] accesses files with S3 tooling.
* [Hadoop Compatible File System][Hadoop] accesses files from Hadoop/Spark/Flink/etc or even runs HBase.
* [Async Replication To Cloud][BackupToCloud] has extremely fast local access and backups to Amazon S3, Google Cloud Storage, Azure, BackBlaze.
* [WebDAV] accesses as a mapped drive on Mac and Windows, or from mobile devices.
* [AES256-GCM Encrypted Storage][FilerDataEncryption] safely stores the encrypted data.
* [Super Large Files][SuperLargeFiles] stores large or super large files in tens of TB.
* [Cloud Drive][CloudDrive] mounts cloud storage to local cluster, cached for fast read and write with asynchronous write back.
* [Gateway to Remote Object Store][GatewayToRemoteObjectStore] mirrors bucket operations to remote object storage, in addition to [Cloud Drive][CloudDrive]
# Architecture#
## Data Lakehouse Features ##
* [S3 Table Buckets][S3TableBucket] expose a dedicated namespace for Iceberg tables with strict layout validation.
* Built-in [Iceberg REST Catalog][IcebergCatalog] runs alongside the S3 endpoint — no external metastore needed.
* Native integrations with [Apache Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], and [RisingWave][RisingWaveIceberg].
* **Master** servers, one or a Raft group of three, track which volume lives on which volume server and hand out file ids. They are not in the read path.
***Volume** servers store blobs in append-only volume files, keep a 16-byte in-memory index per blob, and replicate or erasure-code at the volume level.
***Filer** servers add directories and files on top, with metadata in a store of your choice, and expose HTTP, S3, WebDAV, SFTP, FUSE, and the table catalogs.
The blob store started from [Facebook's Haystack](http://www.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf), erasure coding takes ideas from [f4](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-muralidhar.pdf), and the whole has a lot in common with [Tectonic](https://www.usenix.org/system/files/fast21-pan.pdf) and [Colossus](https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system). How file ids are assigned, written, and looked up, and why a master that tracks volumes scales, is in [Blob Store Architecture][BlobStoreArchitecture]; the services are in [Components][Components] and the [white paper][WhitePaper].
[Back to TOC](#table-of-contents)
## Example: Using Seaweed Blob Store ##
By default, the master node runs on port 9333, and the volume nodes run on port 8080.
Let's start one master node, and two volume nodes on port 8080 and 8081. Ideally, they should be started from different machines. We'll use localhost as an example.
SeaweedFS uses HTTP REST operations to read, write, and delete. The responses are in JSON or JSONP format.
A blob, also referred as a needle, a chunk, or mistakenly as a file, is just a byte array. It can have attributes, such as name, mime type, create or update time, etc. But basically it is just a byte array of a relatively small size, such as 2 MB ~ 64 MB. The size is not fixed.
To upload a blob: first, send a HTTP POST, PUT, or GET request to `/dir/assign` to get an `fid` and a volume server URL:
Now, you can save the `fid`, 3,01637037d6 in this case, to a database field.
The number 3 at the start represents a volume id. After the comma, it's one file key, 01, and a file cookie, 637037d6.
The volume id is an unsigned 32-bit integer. The file key is an unsigned 64-bit integer. The file cookie is an unsigned 32-bit integer, used to prevent URL guessing.
The file key and file cookie are both coded in hex. You can store the <volume id, file key, file cookie> tuple in your own format, or simply store the `fid` as a string.
If stored as a string, in theory, you would need 8+1+16+8=33 bytes. A char(33) would be enough, if not more than enough, since most uses will not need 2^32 volumes.
If space is really a concern, you can store the file id in the binary format. You would need one 4-byte integer for volume id, 8-byte long number for file key, and a 4-byte integer for the file cookie. So 16 bytes are more than enough.
### Read a Blob ###
Here is an example of how to render the URL.
First look up the volume server's URLs by the file's volumeId:
Since (usually) there are not too many volume servers, and volumes don't move often, you can cache the results most of the time. Depending on the replication type, one volume can have multiple replica locations. Just randomly pick one location to read.
Now you can take the public URL, render the URL or directly read from the volume server via URL:
```
http://localhost:8080/3,01637037d6.jpg
```
Notice we add a file extension ".jpg" here. It's optional and just one way for the client to specify the file content type.
If you want a nicer URL, you can use one of these alternative URL formats:
### Rack-Aware and Data Center-Aware Replication ###
SeaweedFS applies the replication strategy at a volume level. So, when you are getting a blob id, you can specify the replication strategy. For example:
When requesting a blob key, an optional "dataCenter" parameter can limit the assigned volume to the specific data center. For example, this specifies that the assigned volume should be limited to 'dc1':
Usually distributed file systems split each file into chunks. A central server keeps a mapping of filenames to chunks, and also which chunks each chunk server has.
The main drawback is that the central server can't handle many small files efficiently, and since all read requests need to go through the central master, so it might not scale well for many concurrent users.
Instead of managing chunks, SeaweedFS manages data volumes in the master server. Each data volume is 32GB in size, and can hold a lot of blobs. And each storage node can have many data volumes. So the master node only needs to store the metadata about the volumes, which is a fairly small amount of data and is generally stable.
The actual blob metadata, which are the blob volume, offset, and size, is stored in each volume on volume servers. Since each volume server only manages metadata of blobs on its own disk, with only 16 bytes for each blob, all access can read the metadata just from memory and only needs one disk operation to actually read file data.
For comparison, consider that an xfs inode structure in Linux is 536 bytes.
### Master Server and Volume Server ###
The architecture is fairly simple. The actual data is stored in volumes on storage nodes. One volume server can have multiple volumes, and can both support read and write access with basic authentication.
All volumes are managed by a master server. The master server contains the volume id to volume server mapping. This is fairly static information, and can be easily cached.
On each write request, the master server also generates a file key, which is a growing 64-bit unsigned integer. Since write requests are not generally as frequent as read requests, one master server should be able to handle the concurrency well.
### Write and Read files ###
When a client sends a write request, the master server returns (volume id, file key, file cookie, volume node URL) for the blob. The client then contacts the volume node and POSTs the blob content.
When a client needs to read a blob based on (volume id, file key, file cookie), it asks the master server by the volume id for the (volume node URL, volume node public URL), or retrieves this from a cache. Then the client can GET the content, or just render the URL on web pages and let browsers fetch the content.
### Saving memory ###
All blob metadata stored on a volume server is readable from memory without disk access. Each file takes just a 16-byte map entry of <64bit key, 32bit offset, 32bit size>. Of course, each map entry has its own space cost for the map. But usually the disk space runs out before the memory does.
### Tiered Storage to the cloud ###
The local volume servers are much faster, while cloud storages have elastic capacity and are actually more cost-efficient if not accessed often (usually free to upload, but relatively costly to access). With the append-only structure and O(1) access time, SeaweedFS can take advantage of both local and cloud storage by offloading the warm data to the cloud.
Usually hot data are fresh and warm data are old. SeaweedFS puts the newly created volumes on local servers, and optionally upload the older volumes on the cloud. If the older data are accessed less often, this literally gives you unlimited capacity with limited local servers, and still fast for new data.
With the O(1) access time, the network latency cost is kept at minimum.
If the hot/warm data is split as 20/80, with 20 servers, you can achieve storage capacity of 100 servers. That's a cost saving of 80%! Or you can repurpose the 80 servers to store new data also, and get 5X storage throughput.
[Back to TOC](#table-of-contents)
## SeaweedFS Filer ##
Built on top of the blob store, SeaweedFS Filer adds directory structure to create a file system. The directory structure is an interface that is implemented in many key-value stores or databases.
The content of a file is mapped to one or many blobs, distributed to multiple volumes on multiple volume servers.
## Compared to Other File Systems ##
# Compared to Other Systems #
Most other distributed file systems seem more complicated than necessary.
@@ -438,9 +271,7 @@ SeaweedFS is meant to be fast and simple, in both setup and operation. If you do
SeaweedFS is constantly moving forward. Same with other systems. These comparisons can be outdated quickly. Please help to keep them updated.
[Back to TOC](#table-of-contents)
### Compared to HDFS ###
## Compared to HDFS ##
HDFS uses the chunk approach for each file, and is ideal for storing large files.
@@ -448,9 +279,7 @@ SeaweedFS is ideal for serving relatively smaller files quickly and concurrently
SeaweedFS can also store extra large files by splitting them into manageable data chunks, and store the file ids of the data chunks into a meta chunk. This is managed by "weed upload/download" tool, and the weed master or volume servers are agnostic about it.
[Back to TOC](#table-of-contents)
### Compared to GlusterFS, Ceph ###
## Compared to GlusterFS, Ceph ##
The architectures are mostly the same. SeaweedFS aims to store and read files fast, with a simple and flat architecture. The main differences are
@@ -469,25 +298,15 @@ The architectures are mostly the same. SeaweedFS aims to store and read files fa
| MinIO | separate meta file per drive for each file | | | Yes | No |
| RustFS | separate meta file per drive for each file | | | Yes | No |
[Back to TOC](#table-of-contents)
GlusterFS stores files, both directories and content, in configurable volumes called "bricks". It hashes the path and filename into ids, and assigned to virtual volumes, and then mapped to "bricks".
### Compared to GlusterFS ###
GlusterFS stores files, both directories and content, in configurable volumes called "bricks".
GlusterFS hashes the path and filename into ids, and assigned to virtual volumes, and then mapped to "bricks".
[Back to TOC](#table-of-contents)
### Compared to MooseFS ###
## Compared to MooseFS ##
MooseFS chooses to neglect small file issue. From moosefs 3.0 manual, "even a small file will occupy 64KiB plus additionally 4KiB of checksums and 1KiB for the header", because it "was initially designed for keeping large amounts (like several thousands) of very big files"
MooseFS Master Server keeps all meta data in memory. Same issue as HDFS namenode.
MooseFS Master Server keeps all meta data in memory. Same issue as HDFS namenode.
[Back to TOC](#table-of-contents)
### Compared to Ceph ###
## Compared to Ceph ##
Ceph can be setup similar to SeaweedFS as a key->blob store. It is much more complicated, with the need to support layers on top of it. [Here is a more detailed comparison](https://github.com/seaweedfs/seaweedfs/issues/120)
@@ -507,13 +326,11 @@ SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Sqlite, Mong
Please note, as Apr 25, 2026 MinIO ceased development. It's strongly discouraged to use that unmaintained software with multiple security bugs. RustFS is a MinIO reimplementation in Rust, Apache 2.0 licensed and still developed, keeping MinIO's storage model down to a byte-compatible on-disk format. So the points below apply to both.
MinIO followed AWS S3 closely and was ideal for testing for S3 API. It had good UI, policies, versionings, etc. SeaweedFS is trying to catch up here.
MinIO followed AWS S3 closely and was ideal for testing for S3 API. It had good UI, policies, versionings, etc. SeaweedFS is trying to catch up here.
The metadata are in simple files. Each file write incurs extra writes to the corresponding meta file, on every drive of the erasure set. Changing only tags or retention rewrites that meta file on all of them, so the write amplification does not shrink with object size.
@@ -530,113 +347,18 @@ There are specific requirements on storage layout, which makes it hard to scale
[Back to TOC](#table-of-contents)
## Dev Plan ##
# Benchmark #
* More tools and documentation, on how to manage and scale the system.
* Read and write stream data.
* Support structured data.
Unscientific single-machine numbers from a MacBook with an SSD. [`weed benchmark`][Benchmarks], 1 million 1KB files, concurrency 16:
This is a super exciting project! And we need helpers and [support](https://www.patreon.com/seaweedfs)!
| | Requests per second | p50 | p99 |
| --- | --- | --- | --- |
| Write | 15,708 | 0.8 ms | 2.6 ms |
| Random read | 47,019 | 0.3 ms | 0.7 ms |
[Back to TOC](#table-of-contents)
## Installation Guide ##
> Installation guide for users who are not familiar with golang
Step 1: install go on your machine and setup the environment by following the instructions at:
Step 3: download, compile, and install the project by executing the following command
```bash
cd seaweedfs/weed && make install
```
Once this is done, you will find the executable "weed" in your `$GOPATH/bin` directory
For more installation options, including how to run with Docker, see the [Getting Started guide](https://github.com/seaweedfs/seaweedfs/wiki/Getting-Started).
[Back to TOC](#table-of-contents)
## Disk Related Topics ##
### Hard Drive Performance ###
When testing read performance on SeaweedFS, it basically becomes a performance test of your hard drive's random read speed. Hard drives usually get 100MB/s~200MB/s.
### Solid State Disk ###
To modify or delete small files, SSD must delete a whole block at a time, and move content in existing blocks to a new block. SSD is fast when brand new, but will get fragmented over time and you have to garbage collect, compacting blocks. SeaweedFS is friendly to SSD since it is append-only. Deletion and compaction are done on volume level in the background, not slowing reading and not causing fragmentation.
[Back to TOC](#table-of-contents)
## Benchmark ##
My Own Unscientific Single Machine Results on Mac Book with Solid State Disk, CPU: 1 Intel Core i7 2.6GHz.
Write 1 million 1KB file:
```
Concurrency Level: 16
Time taken for tests: 66.753 seconds
Completed requests: 1048576
Failed requests: 0
Total transferred: 1106789009 bytes
Requests per second: 15708.23 [#/sec]
Transfer rate: 16191.69 [Kbytes/sec]
Connection Times (ms)
min avg max std
Total: 0.3 1.0 84.3 0.9
Percentage of the requests served within a certain time (ms)
50% 0.8 ms
66% 1.0 ms
75% 1.1 ms
80% 1.2 ms
90% 1.4 ms
95% 1.7 ms
98% 2.1 ms
99% 2.6 ms
100% 84.3 ms
```
Randomly read 1 million files:
```
Concurrency Level: 16
Time taken for tests: 22.301 seconds
Completed requests: 1048576
Failed requests: 0
Total transferred: 1106812873 bytes
Requests per second: 47019.38 [#/sec]
Transfer rate: 48467.57 [Kbytes/sec]
Connection Times (ms)
min avg max std
Total: 0.0 0.3 54.1 0.2
Percentage of the requests served within a certain time (ms)
50% 0.3 ms
90% 0.4 ms
98% 0.6 ms
99% 0.7 ms
100% 54.1 ms
```
### Run WARP and launch a mixed benchmark. ###
`make benchmark` runs [warp][S3Benchmark] mixed S3 traffic against a local `weed server`:
```
make benchmark
warp: Benchmark data written to "warp-mixed-2025-12-05[194844]-kBpU.csv.zst"
Cluster Total: 3302.88 MiB/s, 550.51 obj/s over 43s.
```
Read throughput is bounded by the random read speed of the disks, and grows with every volume server added. More numbers, including multi-node, FUSE, and Hadoop, are in [Benchmarks][Benchmarks], [S3 API Benchmark][S3Benchmark], [FIO benchmark][FIO], and [Independent Benchmarks][IndependentBenchmarks].
[Back to TOC](#table-of-contents)
## Enterprise ##
# Enterprise #
For enterprise users, please visit [seaweedfs.com](https://seaweedfs.com) for the SeaweedFS Enterprise Edition,
which has advanced features, including data recovery, self-healing storage,
For enterprise users, please visit [seaweedfs.com](https://seaweedfs.com) for the SeaweedFS Enterprise Edition,
which has advanced features, including data recovery, self-healing storage,
customizable erasure coding, EC vacuum and repair, etc.
[Back to TOC](#table-of-contents)
## License ##
# License #
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -677,9 +401,114 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
The text of this page is available for modification and reuse under the terms of the Creative Commons Attribution-Sharealike 3.0 Unported License and the GNU Free Documentation License (unversioned, with no invariant sections, front-cover texts, or back-cover texts).
[Back to TOC](#table-of-contents)
# Sponsors #
<h3 align="center"><a href="https://www.patreon.com/seaweedfs">Sponsor SeaweedFS via Patreon</a></h3>
SeaweedFS is an independent Apache-licensed open source project with its ongoing development made
possible entirely thanks to the support of these awesome [backers](https://github.com/seaweedfs/seaweedfs/blob/master/backers.md).
If you'd like to grow SeaweedFS even stronger, please consider joining our
<a href="https://www.patreon.com/seaweedfs">sponsors on Patreon</a>.
Your support will be really appreciated by me and other supporters!
| `recursive` | delete a non-empty directory tree | false; when the filer runs with `filer.options.recursive_delete=true`, deletes are recursive unless `recursive=false` |
| `ignoreRecursiveError` | keep deleting remaining entries after an error | false |
| `skipChunkDeletion` | remove only the metadata, keep volume data | false |
### Tagging
Tags are carried as `Seaweed-`-prefixed request headers, not query
parameters; `?tagging` selects the tagging handler and `?tagging=K1,K2`
lists the keys to remove. Header names are canonicalized on write
(`Seaweed-k1` is stored as `Seaweed-K1`), and the delete list is matched
case-sensitively against the stored names.
```bash
curl -X PUT -H "Seaweed-k1: v1" -H "Seaweed-k2: v2""http://localhost:8888/dir/file.jpg?tagging"
Every image CI pushes to `chrislusf/seaweedfs` and `ghcr.io/chrislusf/seaweedfs` is signed with [cosign](https://docs.sigstore.dev/cosign/verifying/verify/), keyless, by the GitHub Actions workflow that built it, so there is no key to fetch or pin. The signature is attached to the image digest and covers the multi-arch index and each platform image in it; `latest` is the release image under another tag and verifies the same way. Images published before September 2026 predate signing.
cosign prints the digest it verified. Deploy by that digest, or let an admission controller resolve the tag, so what runs is what was checked.
The identity is `https://github.com/seaweedfs/seaweedfs/.github/workflows/<workflow>@<ref>`. The ref is `refs/tags/<version>` for a release and `refs/heads/master` when a variant was republished by hand. The workflow is `container_release_unified.yml` for the release images, `container_dev.yml` for `dev`, `container_latest.yml` for a `latest` rebuilt by hand, `container_release_foundationdb.yml` for the `_large_disk_foundationdb` release image, and `container_foundationdb_version.yml` or `container_rocksdb_version.yml` for the per-version builds. The regexp above accepts release images only; `container_[a-z_]+\.yml@` accepts everything this repository publishes, `dev` included.
The same check as a Kyverno policy, release images only:
`-allowInsecureBind` and bypasses this guard; it leaves the admin API
unauthenticated on the network, so use it only when access is otherwise
restricted (e.g. network policies). The whole `127.0.0.0/8` range and `::1` are
treated as loopback (matching `weed admin`); `localhost` is treated as
non-loopback. Set `admin.ip` to a loopback address only if you also replace the
httpGet probes (e.g. with an `exec` probe that checks `127.0.0.1`).
The `-ip` flag requires SeaweedFS 4.46 or newer; pinning `admin.imageOverride`
to an older image is not supported with this chart version.
### Admin Data Persistence
The admin component can store configuration and maintenance data. You can configure storage in several ways:
@@ -546,6 +570,23 @@ Two things worth knowing before you turn this on:
The DNS selectors default to CoreDNS as kubeadm, kind and the managed offerings from AWS, Google and Azure install it. On OpenShift, override `egress.dnsNamespaceSelector` and `egress.dnsPodSelector` to match `openshift-dns`; see the comment in `values.yaml`.
## Pod and container security contexts
Pod and container security contexts are configurable independently for every built-in workload and remain empty by default for backwards compatibility. The examples in `values.yaml` show how to enable a `RuntimeDefault` seccomp profile, disable privilege escalation and privileged mode, drop all Linux capabilities, and use a read-only root filesystem.
SeaweedFS uses `/tmp` for Unix sockets, temporary uploads, worker task files, and other runtime data. When `readOnlyRootFilesystem` is enabled for a built-in component, the chart mounts a writable `emptyDir` at `/tmp` for its chart-managed containers. Its optional size limit can be configured globally:
```yaml
global:
seaweedfs:
tmpDir:
sizeLimit: 1Gi
```
The chart does not enable `runAsNonRoot` by default because its default `hostPath` storage may be owned by root. To enforce the Kubernetes `restricted` Pod Security Standard, use storage that is writable by a non-root user and configure `runAsNonRoot` or use the OpenShift overrides below.
Security contexts configured for a component also apply to the chart-managed helper containers for that component. User-provided init containers and sidecars must define their own container security context and writable mounts.
## OpenShift Support
SeaweedFS can be deployed on OpenShift or any cluster enforcing the Kubernetes "restricted" Pod Security Standard. By default, OpenShift blocks containers that run as root or use `hostPath` volumes.
@@ -554,6 +595,7 @@ To deploy on OpenShift, use the provided `openshift-values.yaml` which overrides
1. Use `PersistentVolumeClaims` instead of `hostPath`.
2. Enable `runAsNonRoot` and omit hardcoded UIDs to allow OpenShift to assign valid UIDs automatically.
3. Apply appropriate `seccompProfile` and drop capabilities.
4. Use a read-only root filesystem with writable temporary storage at `/tmp`.
{{- if and (not (include "seaweedfs.admin.isLoopbackIp" $adminIp)) (ne $adminAuthEnabled "true") }}
{{- fail (printf "admin.ip is set to %q (non-loopback) but admin authentication is not configured. Since `weed admin` 4.46 refuses to bind a non-loopback address without authentication, the admin container would exit on startup. Set admin.secret.adminPassword or admin.secret.existingSecret, or supply WEED_ADMIN_PASSWORD via admin.extraEnvironmentVars / admin.secretExtraEnvironmentVars, or set admin.ip to a loopback address such as 127.0.0.1 (note: a loopback bind makes the chart's httpGet readiness/liveness probes fail), or set admin.allowInsecureBind to true to opt out via -allowInsecureBind (INSECURE: exposes the admin API unauthenticated on the network)." $adminIp) -}}
{{- end }}
apiVersion:apps/v1
kind:StatefulSet
metadata:
@@ -162,6 +167,7 @@ spec:
-v={{ .Values.global.seaweedfs.loggingLevel }} \
{{- end }}
admin \
-ip={{ .Values.admin.ip | default "0.0.0.0" }} \
-port={{ .Values.admin.port }} \
-port.grpc={{ .Values.admin.grpcPort }} \
{{- if or (eq .Values.admin.data.type "hostPath") (eq .Values.admin.data.type "persistentVolumeClaim") (eq .Values.admin.data.type "emptyDir") (eq .Values.admin.data.type "existingClaim") }}
@@ -170,19 +176,23 @@ spec:
-dataDir={{ .Values.admin.dataDir }} \
{{- end }}
{{- if .Values.admin.masters }}
-masters={{ .Values.admin.masters }}{{- if or $urlPrefix .Values.admin.extraArgs }} \{{ end }}
-masters={{ .Values.admin.masters }} \
{{- else if .Values.global.seaweedfs.masterServer }}
-masters={{ .Values.global.seaweedfs.masterServer }}{{- if or $urlPrefix .Values.admin.extraArgs }} \{{ end }}
-masters={{ range $index := until (.Values.master.replicas | int) }}${SEAWEEDFS_FULLNAME}-master-{{ $index }}.${SEAWEEDFS_FULLNAME}-master.{{ $.Release.Namespace }}:{{ $.Values.master.port }}{{ if lt $index (sub ($.Values.master.replicas | int) 1) }},{{ end }}{{ end }}{{- if or $urlPrefix .Values.admin.extraArgs }} \{{ end }}
-masters={{ range $index := until (.Values.master.replicas | int) }}${SEAWEEDFS_FULLNAME}-master-{{ $index }}.${SEAWEEDFS_FULLNAME}-master.{{ $.Release.Namespace }}:{{ $.Values.master.port }}{{ if lt $index (sub ($.Values.master.replicas | int) 1) }},{{ end }}{{ end }} \
{{- end }}
{{- if $urlPrefix }}
-urlPrefix={{ $urlPrefix }}{{- if .Values.admin.extraArgs }} \{{ end }}
-urlPrefix={{ $urlPrefix }} \
{{- end }}
{{- if .Values.admin.allowInsecureBind }}
-allowInsecureBind \
{{- end }}
{{- range $index, $arg := .Values.admin.extraArgs }}
{{$arg }}{{- if lt $index (sub (len $.Values.admin.extraArgs) 1) }} \{{ end }}
"description":"Jobs currently replicating; pinned at -concurrency means the sync itself is the bottleneck, near zero means caught up or starved by the source (filer.sync)",
"description":"Chunk data carried by the jobs currently replicating; distinguishes workers stuck on a few large files from many small ones (filer.sync)",
Opts::new("SeaweedFS_build_info","A metric with a constant '1' value labeled by version, commit, sizelimit, goos, and goarch from which SeaweedFS was built."),
Opts::new("SeaweedFS_build_info","A metric with a constant '1' value labeled by version, commit, sizelimit, goos, and goarch from which SeaweedFS was built."),
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
hajimohammadi.netandGitHub
Chris LuandGitHub
github-actions[bot]
Oliver RahnerandGitHub
Javier GarciaandGitHub
Alex HuandGitHub
Sascha KühndelandGitHub
Ishwar KanseandGitHub
Tobias GurtzickandGitHub
hsdfatandGitHub
Ilia DemianenkoandGitHub
David ChristopherandGitHub
Feng ShaoandGitHub
Junker der ProvinzandGitHub
yanglongweiandGitHub
Alex KandGitHub
Lorentz KindeandGitHub
Mathieu ArnoldandGitHub
孙超andGitHub
