Sniff identifies TS assets for the coming policy-driven alignment even
though nothing reaches it through repack today. Compile-time assertions
now state each adapter's capabilities explicitly.
Sniff sat in the mandatory adapter interface but has exactly one
caller, the repack gate, which only Indexer adapters can reach - the
hls-ts implementation was dead code. Move it to a Sniffer capability
discovered by assertion like the others: parquet keeps it, hls-ts
drops it, and repack skips the gate when an adapter cannot sniff.
The stored size is uint64; converting a larger value wrapped negative,
sizing the repack sniff buffer with make([]byte, -1) - a handler panic -
and passing Validate a negative size, which means skip the size check.
Refuse repack and answer views stale instead.
The identity digests covered only chunks, while the view's extent path
would serve from inline Content when present - a gRPC update could set
Content with the chunks and size unchanged and segments were served
from bytes the layout never described. Format entries are never written
with inline content, so treat it as disqualifying: views answer stale,
repack rejects it up front, the extent path no longer reads it, and the
source identity digests it so it cannot appear mid-repack unnoticed.
The commit-time check compared only the chunk list, so a concurrent
change that kept the chunks - clearing the TTL, moving the expiry
anchor, hard-linking, going remote - passed verification, and the swap
paired that fresh metadata with chunks uploaded under the old inputs: a
permanent entry pointing at chunks that still expire. Digest everything
the repack consumed - chunk fingerprint, file size, TTL, both time
anchors, the S3-expiry flag, hard-link and remote state - and answer
409 when any of it moved.
The entry lock is filer-local, so a writer on another filer could
commit between repack's read and its swap, and repack then restored the
old bytes over an acknowledged update. Re-read the entry and revalidate
the chunk fingerprint and WORM right before the swap, answering 409 on
any change, and build the new entry from the fresh read so concurrent
metadata-only updates are carried forward. This shrinks the unguarded
window from the whole repack to the commit itself; closing it entirely
needs owner routing.
The layout binding hashed only offset and file id, so a mutation that
kept both - a truncate shrinking chunk.Size, then a sparse extend back
to the original length - passed both the size and fingerprint checks
and served a stale view. Digest size, modification timestamp, cipher
key, compression, manifest status, and SSE type as well.
The fallback capped the remainder at MaxInt32 seconds, which the volume
TTL grid encodes as 68 years - about 35 days shorter than the entry's
lifetime. For the narrow band nothing can round up within int32, store
the chunks without a volume TTL instead: they outlive the entry rather
than predecease it.
SecondsToTTL truncates to the volume TTL grid, so 3599 remaining
seconds became 59m and anything under a minute became no TTL at all -
permanent chunks under an expiring entry. Round the remaining lifetime
up to the smallest representable value instead, and anchor it the way
FindEntry expires entries: S3-expiring entries age from Mtime, others
from Crtime, so a recently overwritten S3 object is no longer treated
as nearly expired.
Also bind each layout to a digest of the chunk list it described.
Offset writes and appends keep Extended while changing the chunks, so
a same-size partial write used to leave the old playlist and extents
being served over new bytes; the views now detect the mismatch and
answer 404 until the file is re-ingested or repacked.
WORM was checked before the entry lock was acquired, so a concurrent
writer could enable it while an ingest, repack, or plain HTTP overwrite
waited, and the commit then replaced a protected entry. Repack now
checks under its lock, and ingest and saveMetaData recheck at commit
time.
gRPC writers, renames, and repack already took the per-path entry lock,
but plain HTTP overwrites committed without it: an overwrite landing
between repack's read and its update was silently replaced, orphaning
its chunks. Take the lock around the saveMetaData and format-ingest
commits, so repack's exclusive hold spans every writer.
Views validated conditional requests against the media entry's ETag,
so re-ingesting identical bytes with a different sidecar changed the
playlist and segment boundaries while clients kept getting 304s. Fold
the encoded layout and the request's view parameters into the ETag the
view serves and checks.
S3 object versions may share one chunk list, so deleting the old chunks
after a repack could corrupt sibling versions; reject those entries
until chunk ownership is tracked.
New chunks also carried the full original TTL, restarting needle
expiry at repack time while entry expiry stayed anchored to creation: a
nearly expired entry left chunks stored for almost a full extra span.
Assign the remaining lifetime instead, and reject entries already past
it.
Ingest admits mediaSequence up to MaxInt64-(count-1), but the payload
decoder rejected anything above MaxInt64-count, so a boundary playlist
ingested successfully and then failed every view. Mirror the ingest
bound, covered by a round-trip test at the boundary.
- namespace the query parameters as format.ingest, format.repack and
format.view, following the mv.from/cp.from dotted convention, so the
general endpoints cannot collide with pass-through client parameters;
requests naming both ingest and repack are rejected
- state Accept-Ranges: none on view responses, which always answer with
whole documents or whole extents
- derive the small-content permission from the boundary source instead
of a second positional bool that a call site could silently swap
- validate the hls-ts layout before returning it, making the formattest
invariant enforced rather than emergent
DecodeLayout rejected names over 256 bytes while Encode accepted them,
so an oversized name encoded fine and then failed every decode. Enforce
the bound in Validate, shared by both directions.
New chunks were assigned with the TTL the request query implied while
the entry kept its own, so repacking a permanent file with ?ttl= made
its chunks expire under permanent metadata. Force the entry TTL onto
the storage option instead.
Filer.UpdateEntry only writes the store, so metadata subscribers never
heard about the new chunk ids while the old ones were queued for
deletion. Emit the update event the way the gRPC UpdateEntry path does.
The cutter materialized every interior cut up front, so a tiny sidecar
declaring one enormous extent could allocate gigabytes of cut offsets
before any media byte arrived. Keep only the extent start offsets,
bounded by the extent count, and derive each cut arithmetically.
Extent chunks are buffered in memory, so an absent limit must not mean
unlimited. Also close the repack chunk reader to release its private
reader cache, and drop the arithmetic capacity hint on the extended-map
allocation.
A merge failure part-way through returned only the flat data chunks,
dropping the manifests already written: cleanup paths could not delete
those needles, and AppendToEntry, which keeps the returned list after
logging the error, lost the wrapped chunks. Return the manifests plus
the not-yet-wrapped remainder instead - a complete representation of
every byte, safe to delete or to keep.
Three hooks, all on the entry's real path so JWT scopes, WORM, and
read-only rules apply unchanged:
- POST /path?format=<name> ingests a multipart index sidecar plus media
and cuts storage chunks on the extents the sidecar declares
- GET /path?view=<name> serves adapter views; rendered documents and
extent streams both ride the normal prefetch path with entry ETag,
preconditions, and HEAD support
- POST /path?repack=<name> derives the layout from the stored bytes and
rewrites the chunks cut on extent boundaries, swapping the entry under
the entry lock and queueing the old chunks for deletion
The layout is advisory: a stale one 404s its views while plain reads
stay untouched. Repack refuses hard-linked, remote, and SSE entries.
The footer already names every row-group byte range, so the adapter
only reads metadata: one extent per row group, the leading magic riding
with the first, and a trailing extent for the page indexes and footer.
Engines that fetch row groups by offset then read exactly the covering
chunks. Parquet needs no view; alignment alone delivers the benefit.
The ingest sidecar is an FFmpeg-style EXT-X-BYTERANGE media playlist;
its segments become the extents and 188 becomes the align quantum, so
every storage chunk holds whole TS packets of one segment. The view
renders a playback playlist with plain numbered segment URLs for
clients that do not speak byte-range HLS, and maps ?seq=N to the
segment's extent. Playlist state the generated playlist cannot
reproduce (EXT-X-KEY, MAP, DISCONTINUITY, GAP, I-frame-only) is
rejected at ingest.
A format adapter reduces one container format to three things the core
understands: extent sizes, an alignment quantum, and an opaque payload.
Capabilities beyond identity (Indexer, SidecarIndexer, Viewer) are
discovered by type assertion. The layout persists in one compact
extended attribute keyed by extent sizes rather than chunk ids, so it
survives chunk manifest folding, and the Cutter turns it into upload
chunk boundaries clamped by maxMB and the align quantum. The formattest
kit holds every adapter to no-panic parsing of truncated input.
The chunking loop always cut at a fixed size. Accept a ChunkBoundaries
source instead, with the fixed size as the default implementation, so a
caller can align storage chunks to structure inside the file. Inline
small-content storage is disabled in that mode because it would drop the
first boundary.
* pb: ask the master what each collection holds
Callers tracking usage were sent every volume in the cluster to add up
themselves, which is the master's largest single allocation.
* topology: summarise what each collection holds
One pass over the topology, allocating per collection rather than per volume.
Regular volumes count once each for logical totals and once per replica for
physical, taken from the lookup index, which is already keyed by volume and so
needs no set of seen ids. Ec shards are node-local so their sizes sum, while
the file and delete counts describe the volume and resolve once every holder
has been seen.
Replicas of one volume disagree while a write is landing or a heartbeat is
late. Walking a full listing took whichever replica the map iteration reached
first, so the answer moved between runs; this takes the largest, which is
stable and never reports usage below what some replica already holds.
* s3: take bucket sizes from the master's summary
The bucket size metrics pulled the whole volume list once a minute and added it
up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six
numbers per collection.
VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire
CollectionStatistics 176 bytes allocated, 47 bytes on the wire
The aggregation moves to the master with it, so the cases the removed tests
covered are now asserted against it directly.
* topology: count the replica holding the most live data
Quotas are enforced on size less deletions, and the replica with the biggest
raw size can be the one that has deleted the most. Counting it reported a
bucket smaller than it is and would leave one writable over its quota, which is
the opposite of what picking the largest was meant to guarantee.
* topology: cap a volume's deletions at what it holds
Live usage is read as a collection's size less its deletions, so a volume
reporting more deleted bytes than it has cancels live bytes belonging to other
volumes in the same bucket and reports it smaller than it is. Replica selection
already floored that volume's own live size at zero; the totals have to agree
with it.
* storage: share the volume strings a cluster repeats
Decoding a heartbeat allocates a fresh string for the collection, disk type and
remote backend of every volume, and a master holding a million volumes then
holds a million copies of the same handful of names.
Not the remote storage key, which is unique per volume: interning that would
fill the table rather than share anything.
800k volumes registered from a heartbeat that has actually been over the wire:
227 -> 211 B/volume, and 238 -> 214 when the volumes are tiered, since the
backend name shares too.
* storage: hold the interned strings rather than let them be collected
unique.Make clears its entries by weak reference, and its canonical value does
not survive a collection even while a caller still holds the string it handed
back -- so a volume reported later would get a second copy of a name the rest
of the cluster already shares. With only changed volumes reported, most are
interned once and never again, so that is the common case rather than a corner.
The table therefore only grows, which is why it stays restricted to values
drawn from a small set. Ten thousand collections keep a few hundred kilobytes.
* iceberg: stamp a default name mapping on new tables
* iceberg: repair non-compliant manifests at commit
* s3tables: verify ClickHouse writes read back through PyIceberg
* iceberg: carry the manifest-list content into repaired manifests
* iceberg: refresh the default name mapping on schema evolution
* iceberg: merge historical names into the refreshed name mapping
* iceberg: never fail a commit on repair fallout
* iceberg: harden manifest repair against writer dialects
* s3tables: keep PyIceberg reader stderr out of row data
* iceberg: keep name mappings unambiguous across field id reassignment
* iceberg: align existing manifest content metadata with the list entry
* s3: add the RenameObject endpoint
PUT /{bucket}/{key}?renameObject with x-amz-rename-source moves an object
through the filer's AtomicRenameEntry, so no bytes are read or rewritten and
the ETag, tags and SSE keys travel with the entry.
Only unversioned buckets: a versioned rename would have to rebuild the
.versions chain, and AWS offers RenameObject on directory buckets, which
cannot be versioned. The source arrives in a header, so it is authorized
separately for read and delete; both keys are locked, in key order, across the
precondition checks and the move.
* s3: let a matched source ETag precondition settle its date precondition
RFC 7232 has an ETag precondition outrank the date precondition on its own
side, and AWS documents the same for CopyObject: a matching
x-amz-copy-source-if-match with a failing x-amz-copy-source-if-unmodified-since
copies rather than returning 412. The source check evaluated all four headers in
sequence, so the date header could still veto a decided ETag match.
validateConditionalHeadersForReads already applies this precedence; the source
path now matches it.
* s3: cover a rename source named as a directory without a trailing slash
Renaming a directory would move a whole subtree, so it has to stay a missing
key whether or not the caller wrote the trailing slash.
* s3: accept a bare object key as the RenameObject source
AWS spells x-amz-rename-source both ways. Its CLI, Java and Rust examples pass
the bare source key, and only a second CLI example and the boto3 conditional
example pass bucket/key; the API reference's own example is a bare key too. The
header was read as bucket/key only, so the form AWS leads with was rejected with
InvalidArgument and the endpoint was unusable as documented.
A value is now read as a literal key first — the only reading that can never
name the wrong object — and as bucket-qualified second, when it carries the
request's own bucket and the literal key does not exist. That costs one extra
lookup only for a source that starts with the bucket's own name.
Another bucket's name in the source is no longer a distinct error: RenameObject
moves within one bucket, so it is simply part of a key this bucket does not
hold, and it reports NoSuchKey.
* s3: only a proven absence picks the other reading of a rename source
A source that resolves to a directory is not a miss to fall through on: the
literal path is still what the caller named, so answering for it beats renaming
a different object under the bucket-qualified reading. With a directory at
bucket/source.txt and an object at source.txt, a rename naming the former moved
the latter.
A failed lookup is not a proof of absence either, so a blip can no longer
redirect a rename to the other reading.
* topology: keep per-node volume state with the location it describes
The read-only and oversized indexes were maps from volume id to a list of the
nodes reporting that state -- the same key space the lookup index already
holds, kept a second and third time. Nothing ever asked which nodes; both are
only ever asked whether any node does.
So the state rides on the location list as a bit per entry, and the two indexes
go. Removing a location shifts the bits with it, and a node replacing another
at the same address inherits its slot, since that is what happens to the
location too.
800k volumes, 90% read-only: readonly index 40.0MB -> 0, lookup index
42.5MB -> 48.6MB for the bits, 33.9MB net.
* topology: rebuild the location flags when stale entries are dropped
Refresh rebuilds the location list, so leaving the flags alone left bits
describing whoever moved into the dropped entries' place.
* topology: assert the refreshed flag survived, not just that it moved
Clearing the mask rather than rebuilding it would have passed: the check that
the flag lands on the right location is done by clearing it, which an already
empty mask satisfies.
* topology: track volume size only where writes can land
Size tracking decays pending assignment estimates so the master does not
overfill a volume before heartbeats catch up. Nothing is ever assigned to a
read-only volume, so an entry for one can never be consulted -- and in a tiered
cluster that is most of them, which made this the volume layout's largest cost.
A volume held out of the writable list for capacity is not read-only and keeps
its entry: that entry is what enforces the recovery delay.
800k volumes, 90% read-only: sizeTracking 79.1MB -> 8.4MB, and the crowded set
falls out with it because a read-only volume no longer reaches the threshold
check at all.
* topology: decide size tracking per volume, not per reporting replica
A volume is unwritable if any replica is read-only, so asking the replica whose
heartbeat happened to arrive made the answer depend on arrival order: a
writable replica reporting after a read-only one put the tracking back.
Ask the volume instead, which also drops the caller-supplied flag and the churn
it caused. The crowded entry goes with the tracking, since leaving it behind
would only move the memory this releases.
Costs a map lookup per replica on a full-list heartbeat, about 19ms per 100k
volumes and no allocations, on a path that is now rare.
* s3api: fix ListObjectsV2 dropping objects under a partial prefix
ListObjectsV2 with a prefix that names only part of a directory name -
"data/a" matching both "data/a/..." and "data/ab/..." - dropped objects
when paginating with a small max-keys, reporting IsTruncated=false before
all matching keys had been returned.
On resume, normalizePrefixMarker took the listing directory from the
continuation marker ("data/a/") and discarded the "a" name prefix, so once
the marker's subtree was exhausted the sibling directories that also
matched the prefix were never listed.
Resolve the listing directory and name prefix from the prefix instead of
the marker, so siblings sharing the name prefix are still listed after the
marker's subtree.
Fixes#10652
* s3api: cover a sibling-directory marker in the partial prefix tests
* s3api: resume a truncated delimited listing from the emitted CommonPrefix
Rebuilding the continuation token from the request dir plus the listing prefix
assumes that prefix names a directory. With a partial name prefix like "data/a"
it produced "data/a/ab/" for the sibling prefix "data/ab/", so the next page
resumed in the wrong subtree and repeated it forever.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* heartbeat: name departed volumes in delta heartbeats
* master: release the lookup index with a deleted collection
* master: keep a fresh grow safe from the report that raced it
* volume: name the volumes a deleted collection took with it
Deleting a collection left the master to work out what went by omission from
the next full volume list, which it no longer gets: heartbeats carry the whole
list only when the master asks for it. The volumes a bucket's churn creates and
destroys between two of those requests are never named in either direction, so
the master keeps counting their slots as occupied and a cluster that creates
and drops collections quickly runs its free-slot accounting dry -- assigns fail
with no free volumes left while the disk holds a handful of volumes.
The destroy path already knows exactly which volumes it removed, so send them
down the same channel every other deletion uses.
* rust: name the volumes a deleted collection took with it
Mirrors the Go volume server. The notify path derives its deltas by diffing
snapshots, so a collection delete that does not wake it is invisible until the
master next asks for the whole list.
* topology: mark a volume crowded only if it can take writes
Crowding asks for more room to write into, and the writable-volume refresh loop
marked anything past the threshold regardless of whether writes could land
there. Growth already discounts those by intersecting the crowded set with the
writable list, so the entries changed no decision and only took space -- in a
tiered cluster, one for nearly every volume.
* topology: wait for the crowded-volume collector before reading what it saw
Closing the stop channel does not order the collector's writes against the
test's reads.
* topology: drop the sleep from the crowded-volume test
The channels are unbuffered, so every send has been received by the time the
sweep returns, and waiting for the collector covers the recording. The sleep
only suggested the result turned on timing.
* filer: keep the existing peer subscription on a repeated add
A cluster node add for a peer that is already followed restarted the
subscription, dropping the metadata events between the two runs.
* master: tell a connecting client the current cluster membership
Cluster node updates are only broadcast to the clients connected at that
moment. A filer that lost its master stream while a peer came back never
learned about the peer, and stopped replicating its metadata for good.
* test: a filer joining the master learns about the filers already there
* test: a filer resubscribes to a peer that registered while it was disconnected
Runs the reported sequence against real processes: filer2 leaves, filer1
is paused and its master stream is broken, filer2 registers again, and
filer1 has to replicate from it after reconnecting.
* volume: validate the file extension in CopyFile and ReceiveFile
CopyFile and ReceiveFile build an on-disk path from the client-supplied
Ext. Both are intentionally ungated for cluster-internal peers, so a
value like "/../../x" is joined onto the volume directory and, once
path-cleaned, resolves outside it -- an EC-shard receive can then write,
and CopyFile read, anywhere the process can reach.
Constrain Ext to a real suffix (a leading dot followed by alphanumerics)
before it is used to build any path, so it can no longer carry a
separator or a parent reference.
* test: use an alphanumeric missing-file extension in the copy variants
The not-found and stop-offset-zero cases used ".definitely-missing" as a
deliberately absent source. The extension is now validated, and the hyphen
makes it invalid, so switch to ".missing" -- still a nonexistent file, but a
real extension shape.
* volume: validate the collection in CopyFile and ReceiveFile
The client-supplied Collection is folded into the on-disk path as
"<collection>_<vid>" by VolumeFileName and EcShardBaseFileName, both joined
with path.Join / util.Join. A Collection carrying a separator, e.g.
"../../x", therefore path-cleans to a target outside the volume directory,
the same escape the extension check just closed. Reject a collection that is
a bare parent reference or holds a separator; ordinary names ('.', '-' and
all) still pass.
* filer: drain pending log chunk refs when the metadata stream ends
In metadata chunks mode the server sends log file refs in responses of their
own, and the client can only read them once it knows the run of refs is over.
That was inferred solely from the arrival of a normal event, so refs still
pending when the stream ended were dropped: the subscription returned no
events and no error.
A follower never noticed, because it runs forever and a live event always
arrives to close the run. A bounded subscription — StopTsNs set, range already
in the past — can receive nothing but refs and then EOF, and silently reports
that nothing happened. For anything auditing a path that is the worst possible
answer, since an empty result is indistinguishable from a quiet period.
Drain on EOF as well as at the transition point.
* filer: add filer.meta.scan to audit one directory's change history
Reconstructing what happened to a path means replaying the metadata log, and
filer.meta.tail is built for watching rather than auditing: it follows forever
unless given a stop, prints multi-line JSON, and takes ranges only as durations
before now, so an incident timestamp has to be converted by hand.
Its -pattern also cannot find a versioned object. A versioned key is stored as
<key>.versions/v_<id>, so the events carry the names "<key>.versions" and
"v_<id>" and a pattern of the object's own name matches neither — the search
comes back empty while the object is being written continuously.
filer.meta.scan prints one line per change, stops at the end of the range,
accepts absolute -since/-until with an explicit -tz, and reports versioned
writes against the object key with the version id alongside, so -name matches
the key a client would ask for. Delete markers are labelled as such rather than
appearing as zero-length writes, and pointer flips on the .versions container
are distinguished from writes of object data.
* filer.meta.scan: read persisted log chunks from the volume servers
Reading a range through the filer makes it decode every log entry in that
range and filter each one, so the cost lands on the filer and does not shrink
when the prefix is narrow — only the bytes on the wire do. On a cluster whose
metadata log is dense that is the expensive part of a scan, and it is charged
to the process least able to spare it.
Enable metadata chunks mode: the filer hands out log chunk ids and the scan
reads them from the volume servers itself. ReadLogFileRefs re-applies the same
path filter client-side, so the output is unchanged — verified identical to
the filer-read path over the same range, including after a restart drops the
in-memory buffer and the data must come off disk.
Direct read needs a route to the volume servers that the filer does not, so a
failure before anything has been printed retries through the filer; retrying
after partial output would duplicate lines. -directRead=false forces it.
* filer.meta.scan: confirm an empty direct-read result through the filer
An audit that returns nothing is read as "nothing happened here", so it is the
one answer that must not be produced by a bug. Direct read has more ways to
come back empty than the filer path does — it needs a route to the volume
servers, and it depends on the ref-drain contract holding.
When direct read yields no changes, re-run through the filer before reporting
it, and warn if the two disagree. Re-running is safe only because nothing was
printed; after partial output a replay would duplicate lines instead, so that
case reports the error rather than retrying.
In metadata chunks mode the server sends log file refs in responses of their
own, and the client can only read them once it knows the run of refs is over.
That was inferred solely from the arrival of a normal event, so refs still
pending when the stream ended were dropped: the subscription returned no
events and no error.
A follower never noticed, because it runs forever and a live event always
arrives to close the run. A bounded subscription — StopTsNs set, range already
in the past — can receive nothing but refs and then EOF, and silently reports
that nothing happened. For anything auditing a path that is the worst possible
answer, since an empty result is indistinguishable from a quiet period.
Drain on EOF as well as at the transition point.
* master: evict a dead peer via the local raft handle
OnPeerUpdate only runs on the leader, and the AddVoter branch right above
mutates the local raft directly. The remove branch instead dialed our own
RaftRemoveServer back over gRPC. Drop the self-dial and remove the peer
through the local handle, matching the add path. This also leaves operator
tooling as the only caller of the RaftRemoveServer RPC.
* master: require whitelist auth for raft membership RPCs
RaftAddServer, RaftRemoveServer and RaftLeadershipTransfer rewrite raft
quorum but had no caller check beyond "am I the leader". Any client that
could reach the master gRPC port could add an unreachable phantom voter
and stall the write path.
Gate the three on the admin whitelist, mirroring the volume server's
checkGrpcAdminAuth. With no whitelist configured the guard allows every
caller, so default and single-master deployments are unaffected;
operators who set -whiteList get these RPCs locked down to it. The
leader's own dead-peer eviction no longer dials these RPCs, so the only
remaining callers are operator tooling.
* pb: let a heartbeat carry only the volumes that changed
A partial list cannot travel in volumes: a master that did not understand it
would read the absences as deletions. So changes get their own field, used only
once the master has said it compares digests and can tell when it has fallen
behind.
* master: apply the volumes a heartbeat reports as changed
Only the named volumes are touched. A full report says the server holds exactly
these; a changed report says nothing about the ones it leaves out, so absence
must not read as removal.
Also advertises that the master compares digests, which is what lets a server
stop sending its whole list. Advertising it once per connection means a server
reconnecting to a master that does not is back to full lists straight away.
* volume: send only the volumes that changed once the master accepts them
The whole list goes on every heartbeat until the master says it compares
digests, and again whenever it asks, so a master that cannot tell when it has
fallen behind never has to.
has_no_volumes stays derived from a full list alone. Deriving it from what a
heartbeat happens to carry would make a quiet one read as a server that had
lost every volume, and the master would drop them all.
The digest still covers every volume held rather than the ones sent, which is
what lets the master confirm that applying the changes left it current.
Reporting state is per-connection: a server that reconnects, or reaches a
different master, starts again from the full list.
* volume: let the zero reporting state stand for having told no master anything
A Store built as a literal, which tests do, left the reporting state nil and
panicked on the first heartbeat. As a value its zero form already means nothing
has been reported to anyone, which is exactly the state that sends the whole
list.
* rust: send only the volumes that changed once the master accepts them
Mirrors the Go volume server, with one hazard the Go side does not have: mount
and unmount deltas here are derived by diffing successive heartbeats, so a
heartbeat that carries a partial list would report every volume it left out as
unmounted. Collecting now returns the full set alongside the message, and every
site that diffs uses that rather than what went on the wire.
* volume: do not let a full-list request be lost to the heartbeat it raced
The request arrived while a heartbeat was already being built as a delta, and
committing that heartbeat cleared it, so the master waited for another digest
mismatch before asking again. Count the requests and clear only the one the
heartbeat answered.
* rust: stop marking volumes reported by a heartbeat that is thrown away
The state-notify path collected a heartbeat only to diff its volume list, then
sent a delta message of its own and dropped the one it had collected. Once
collecting recorded what the master had been told, every mount or unmount
silently marked the changed volumes as sent, and the master learned of them
only after a digest mismatch.
Snapshotting no longer records anything, and no longer expires ec volumes
whose deletion that path was already discarding.
* master: announce only the volumes a change actually brought
Every changed volume was broadcast as a new location. Volumes grow constantly
and growth moves no location, so on a busy cluster that told every connected
client about volumes it could already reach, filling bounded broadcast queues
and pushing out the topology updates that matter.
* master: ask for the full list when only one can repair the master
Delta heartbeats stop the full report, and with it the only thing that
re-registers a volume the lookup index lost. The volume server cannot see that
divergence and its digest cannot show it, so the master now checks its own two
indexes agree and asks for the list when they do not.
A node reporting one volume id twice is kept on full lists for the same reason
rather than merely skipped: its digest can never be verified, so nothing else
would tell the master what it had stopped holding.
* master: keep the volume options on every heartbeat response
A volume server takes them from whatever response arrives, and preallocate is a
bare bool with no way to tell off from unmentioned. A response sent to ask for
the volume list therefore turned preallocation off until the server reconnected.
Responses sent mid-stream now start from the configured options rather than
being built field by field.
* master: announce a volume the lookup index had lost
Repairing the index makes the volume servable again, but clients were told it
went when the node dropped out and nothing told them otherwise: the disk map
still held it, so it did not count as an arrival.
Reaching the lookup index is what makes a volume servable, so recovering an
entry there is an arrival as far as clients are concerned, on both the full
report and the changed-volume path.
* s3tables: add ClickHouse iceberg catalog integration test
* ci: run the ClickHouse iceberg catalog test
* s3tables: bound setup HTTP calls in the ClickHouse test
* s3tables: pin the ClickHouse writer image dependencies
VolumeLocationList.Stats subtracts the deleted figures from the totals to
report live size and needle count. Both deleted figures are maintained as
counters independent of the totals they come off, so either can transiently
exceed its total, and neither subtraction was clamped.
Unclamped, the size wraps to ~16 EB. The count is signed so it merely goes
negative, but VolumeLayout.Stats converts it with uint64(fileCount), which
turns it into ~1.8e19 just the same. Either one swamps the cluster totals
behind /dir/status, /vol/status and Topology.CollectionVolumeStats.
commandFsMergeVolumes.getVolumeSize had the same unclamped subtraction, where
a wrapped size reads as a volume far too large to join any merge plan.
Clamped to zero, matching the guards already in CollectionInfo.LogicalSize
and the admin server's logical-size accumulator.
WriteFile's own existence probe runs while the file does not exist, and
WinFsp may serve that answer from its metadata cache for up to the
mount's FileInfoTimeout. A reader racing into that window failed its
open with not-found, which is the cache being a cache, not a defect in
concurrent reading. Establish visibility once before racing the readers,
so the test exercises what it is named for.
* wdclient: keep the location of a volume reported added and removed at once
A volume moved between a server's disks arrives in both lists of one message,
and the server still has it. Additions were applied before removals, so the
removal won and the client was left with no location for a volume that never
went anywhere.
Reordering would swap the bug for a window where the volume resolves nowhere,
since the two updates take the lock separately. Skip the removal instead, so
the order the lists are applied in stops mattering.
* wdclient: build each ec update explicitly in the move test
Reusing one response object and adding the deletion to it left the overlap the
test turns on implicit, and reading it as a delete-only update is the natural
mistake.
* topology: follow a volume that moved between a server's disks
The heartbeat diff asked only whether a volume id was reported anywhere on the
node, so a volume that moved to a disk of another type stayed on the disk it
left as well. The master then held two copies of it forever: the volume count
was overstated, and GetVolumesById returned whichever disk the map iterated
first, so lookups could hand back the disk the volume had already left.
Track which disk types the heartbeat named each volume on, and treat a volume
named on another disk as absent from this one. Disk types are interned to an
index because a server reports a handful of them across hundreds of thousands
of volumes.
A volume named on two disks at once is a stale twin rather than a move, and is
still kept on both -- dropping one would tell the master a replica vanished.
Only a volume named twice on one disk type is unrepresentable, so that is now
what marks the node, rather than any repeat of an id.
* master: do not tell clients a moved volume left the node
A volume moved between a node's disks is removed from one and added to the
other, so it lands in both lists of the same heartbeat. Clients apply additions
before deletions, so the removal wins and they end up with no location for a
volume that never went anywhere.
Skip removals for volumes the node still holds, as the ec shard paths already
do, and update the topology before judging the delta removals so an unmount
that really did happen is still reported.
* trim the comments on this change to the parts that are not evident
* master: judge a volume removal on normal replicas alone
HasVolumesById answers for ec shards as well, so a replica encoded into ec
shards looked like it was still on the node and clients were never told the
normal location had gone. They hold normal and ec locations separately and
prefer the normal one from the same generation, so that location would have
gone on shadowing the shards.
Every enumeration of a directory walked the whole FUSE machinery, so
reopening a folder cost what opening it did. The kernel has a cache for
exactly this: with FOPEN_CACHE_DIR the listing lives in the directory's
page cache and a repeat enumeration never reaches the mount at all.
Local mutations already drop that cache in the kernel. Remote ones
arrive through the metadata subscription, so the entry invalidation
worker now also tells the kernel which directory changed. The worker is
the one place this is safe from: notifying from a thread serving a
kernel request can deadlock against the page it holds, which is why the
file paths deliberately avoid InodeNotify.
Measured in a Linux container, 20k-entry directory, ls repeated:
warm listing before 199-355ms after 6-9ms
A file written from outside the mount appeared in the next listing
within a second, through the subscription notify, and the listing
re-cached after.
The memory is the kernel's page cache: reclaimed under pressure, owned
per-directory, and covering read-through directories the mount-side
caches never see.
* mount: read oversized directories through instead of caching them
Visiting a directory pulls every child from the filer into the local
LevelDB before the first listing returns. For a directory of a few
million entries that is minutes of streaming, gigabytes of local store,
and gigabytes of decoded entries in flight -- paid by a mount that may
only walk the directory once.
A build that crosses -cacheDirMaxEntries (default ten thousand) now
stops, cleans up, and marks the directory read-through: listings stream
from the filer with pagination, the way update-hot directories already
do, and lookups in it consult the filer per entry as any uncached
directory does. The refusal is remembered, so the next visit fails fast
instead of streaming to the limit again, and an oversized ancestor is
stepped over when caching its subdirectories rather than wedging every
listing beneath it.
The direct path keeps the same pagination state on the handle, so a walk
that crosses the limit mid-flight carries on from where the cached walk
reached.
* mount: an ancestor found oversized must not fail its descendants
Visiting a directory builds its whole uncached ancestor chain in one
group, so the first discovery that an ancestor is oversized cancelled the
group and surfaced as the listed directory's own refusal: the descendant
build was aborted and the caller marked the descendant read-through,
leaving a perfectly cacheable directory streaming from the filer until
its inode was forgotten. The earlier test missed this by pre-marking the
ancestor, which exercises only the fast path.
The refusal of any directory other than the one being listed is now kept
out of the group's result; it is already remembered for the next visit.
collectVolumeStats walked only DataNode.GetVolumes(), which returns the
regular volumes on each disk. An encoded volume leaves that set and is
reported through GetEcShards instead, so total_disk_bytes and
total_volume_count silently excluded every erasure-coded volume: a cluster
that encoded everything reported zero bytes and zero volumes while still
counting as a volume server.
Sum each holder's shard sizes into the byte total, parity and extra copies
included, matching how a replicated volume's used size counts every replica
and how CollectionEcVolumeStats already reports EC footprint. Count volume
ids rather than shard entries, since one volume's shards are spread over
many nodes and would otherwise multiply the volume count by the number of
holders.
* pb: carry a volume digest on the heartbeat
The full volume list is the only way a master notices a volume that vanished
without a delta, so it cannot simply be dropped. A digest gives the same
guarantee without the list, and a way back to the list when they disagree.
The digest has explicit presence: a server holding no volumes reports 0, which
has to stay distinguishable from a server that does not compute one at all.
* volume: report a digest of the volumes each heartbeat carries
Digests exactly what goes on the wire: volumes skipped as quarantined, phantom
or expired are absent from both the list and the digest, so the master compares
against the same set the server meant to report.
Runs the master's own hash over the master's own conversion of the message, so
the two ends cannot drift into disagreeing about a field.
* master: check the reported volume digest and ask for the list on a mismatch
Compared after everything the heartbeat carried has been applied, so agreement
means the master is current rather than that nothing changed.
Servers reporting no digest are untouched, and a mismatch on a heartbeat that
already carried the full list is reported rather than answered: there is
nothing further to ask for, so asking again would loop. Nodes reporting one
volume id twice are skipped for the same reason.
* rust: report the heartbeat volume digest
Mirrors the Go volume server. The master compares this against a digest it
computes itself, so the hash has to agree byte for byte across the two
implementations, not merely be a hash of the same fields: report_hash_vectors
pins it against values generated by the Go side, and the ttl and replica
placement narrowing the master applies when it decodes a message is applied
here too rather than assumed away.
A drift there would not corrupt anything, but every volume server on this
implementation would report a digest the master can never match and fall back
to sending its whole volume list forever, which is the cost the digest exists
to avoid.
* master: pin what the digest check does to each kind of report
The upgrade story rests on these: a server that reports no digest is never
asked for anything, so the two sides can be upgraded in either order, and a
disagreement that resending cannot fix is reported rather than re-asked, so it
cannot loop.
* topology: enumerate the digest coverage test from the message
The list of fields was written out by hand, so a field added to
VolumeInformationMessage later would fall outside the digest while the test
went on passing, and a change to it would never reach the master. Walk the
message descriptor instead.
Some fields are narrowed or normalised on the way into VolumeInfo, so the
smallest change to the wire value can land back on the stored one; the test
offers several values per field and asks only that some change is visible.
* telemetry: put the version pie back, beside the stacked chart
The two answer different questions and the pie was the better answer to one
of them: what the fleet is on right now, at a glance. Restore it under its
old name and give the stack its own card as Versions Over Time, so the pie
is the last day of the chart below it.
* telemetry: draw the distribution pies at the size of their cards
Both pies kept the canvas tag's 2:1 ratio at the card's full width, so they
came out around 560px tall and spilled past the card they sit in. Give them a
height to fill instead, and build every chart after the dashboard is shown: a
canvas in a display:none container measures zero, and a pie sized from that
never grows back.
* topology: digest the volumes a master believes each node holds
A volume server resends its whole volume list every heartbeat because that list
is the only way the master can notice a volume that vanished without a delta.
A digest gives the master the same guarantee without the list: the two ends
agree iff the master's copy is current.
VolumeInfo.ReportHash covers every field of VolumeInformationMessage, so a
change the hash misses is a change the master would never hear about. Both ends
run it over the same converted VolumeInfo, so they cannot drift apart.
Disk keeps the xor of its volumes' hashes, which is order-independent and its
own inverse, so add, update and remove each stay O(1) and the running value
needs no per-volume storage.
Nothing reads the digest yet; the heartbeat protocol change comes next.
* topology: test that a changed-volumes-only heartbeat reconciles
The digest is not a change detector -- in a live cluster some volumes always
have changed. It answers whether the master holds what the volume server holds
once the heartbeat's own changes are applied, so reporting three volumes out of
fifty has to reconcile while a volume lost without a delta must not.
* topology: digest the lookup index too, not just the disk maps
The reported digest answers whether the master holds what the volume server
holds. It cannot answer whether the master can serve those volumes: the disk
map and the lookup index are maintained separately, and a disconnect racing a
reconnect drops a volume from the index while leaving it on the node. The
server's report is identical either way, so a digest built from the disk maps
alone matches while the volume answers 'volume id not found'.
Track a second digest over volume ids on both sides of that split, so the
master can see its own indexes disagree without the volume server's help, and
without the O(volumes) scan the full heartbeat currently relies on.
* topology: exclude nodes reporting a duplicate volume id from the digest
A volume id can end up mounted on two disks of one server -- a stale twin
re-attached after a disk repair, which the store handles rather than rejects.
The server reports both copies with different disk ids, but the master keys
volumes by id alone within a disk type and keeps only the last one. Its digest
can then never equal the server's, and no amount of resending the full list
would fix it.
Detect it from the report itself, where deduplicating the ids already tells us
the count, and mark the node. A marked node has to keep sending full lists;
representing both copies is a separate question, and nesting the volume map by
disk id would cost more memory than the digest saves.
* topology: move the lookup digest with the entry, not the node passed in
Two volume servers can hold one address: GetOrCreateDataNode keys on the id a
server reports and refuses to merge a new id onto an address an older node
still claims, while the lookup list keys on address alone. Registering the
second server therefore displaces the first from the entry, and unregistering
through either removes whichever node the entry named.
Crediting the node handed to Set and Remove instead of the one actually
displaced or removed left the digest on the wrong node. A displaced node went
on reporting a consistent index while it could no longer serve the volume,
which is exactly the silent unavailability the digest exists to catch.
Set and Remove now return the node they displaced and removed, so ownership
can be transferred rather than assumed.
* mount: size the direct listing slice from the batch, not the offset
The limit passed here is skipCount+batchSize when a client resumes a
fresh handle partway through a directory, so preallocating for it turns
the client's cookie into an allocation: a readdir at offset 3,000,000
reserves 24MB before the first entry arrives, and an offset near the
uint32 ceiling asks makeslice for ~4.29e9 elements.
* mount: stop replaying a directory that shrank past the resume offset
A client that opens a fresh handle and resumes at a cookie from an
earlier, larger listing gets a preload that cannot reach the entry before
that offset. The resume name was then left empty and the follow-up batch
listed from the directory's first child again, so the client was handed
every name a second time.
The stream always runs from the first child, so failing to reach that
entry means the directory is simply shorter than the offset. That is the
end of it.
* mount: page a directory from where the store reached
The batch loader treated a short batch as the end of the directory, but
the meta cache drops an expired child after the store has already spent
it against the limit, so a batch that filled up could still deliver fewer
entries than asked for. A directory with a handful of expired children
would stop listing early and hide every child behind them; a whole batch
of expired ones truncated the listing to nothing.
ListDirectoryEntries now reports the name the store itself reached. That
is both the sound end-of-directory signal -- the store returning nothing
-- and the right cursor, since resuming from the last visible name would
re-read the dropped children every round and never get past a batch that
was entirely expired.
* mount: drop the entries a directory walk has already passed
entryStreamOffset was only ever written by reset, so the stream a handle
holds grew for the life of the walk and a directory was retained whole
even though nothing could read the entries behind the client's position
again. A 10M-entry walk parked millions of entries per handle, and
NFS-Ganesha opens several on the same directory.
Offsets index into the stream from entryStreamOffset, so advancing the
two together keeps them lined up. One entry is held back because the next
batch resumes from the name immediately before the offset. Seeking back
behind what is still held now restarts the directory, which is what the
offset scheme can honestly support -- it previously returned nothing.
* telemetry: keep the reported version in the daily history
The version only ever lived on the instance record, which holds a cluster's
latest report, so there was no way to ask what anything ran last Tuesday.
Record it per sample, and let the daily axis carry strings as well as counts.
State written before this has no version on its samples. The newest sample is
the report the instance record itself came from, so fill that one in on load
rather than starting a version series a day late.
* telemetry: serve the fleet's version make-up over time
/api/versions gives how many clusters ran each release per day, on the same
axis and hold-forward rule as the cluster sizes. Releases are ordered by
number rather than by size: the caller stacks them, and a stack whose order
changes with the counts is unreadable over time. The tail past the limit is
summed into "other" so the stack still adds up.
Days with no version are dropped before the axis is built, so the series
spans the days it knows a version for instead of climbing out of blanks.
* telemetry: draw version distribution as a stacked growth chart
The pie only ever showed today. Stacked over 30 days the height is the
confirmed fleet and each band is a release, so one chart carries the growth
and the rollouts at once. Newest release on the floor, so the band being read
is anchored to the axis instead of riding on everything below it.
Eight fixed hues instead of the evenly spaced ones the cluster stacks use:
evenly spaced put a green and a cyan close enough to be hard to tell apart,
which matters for a set you read rather than a wall of anonymous ids.
Versions past the eighth fold into "other", and each band carries its own
number so the chart reads without matching colours against the legend.
* filer: decode a listed entry without building its chunk list
A readdir reads attributes and never looks at chunks, but decoding an
entry builds the whole chunk list first: four allocations per chunk, all
of it thrown away. On a directory of ordinary 4MB-chunked files that is
most of what listing costs.
DecodeAttributesOnly walks the wire format and hands everything except
the chunks to the generated unmarshaller, so new fields in filer.proto
need no attention here. The chunks are still measured, because the S3
copy and multipart paths deliberately store a zero FileSize and let the
chunk extents define the size, but nothing is allocated to do it.
The blob is only re-encoded once a chunk is actually seen, so an entry
without any -- every directory, for one -- is unmarshalled where it lies
and pays nothing for the walk.
Listings opt in through the context, the way the lazy remote paths
already do; a store that ignores it stays correct.
chunks full attrs-only allocs
0 312.8n 310.1n ~ 1 -> 1
1 686.1n 411.1n -40.07% 7 -> 1
4 1.742u 667.4n -61.69% 24 -> 1
16 5.770u 1.544u -73.25% 86 -> 1
64 25.23u 6.004u -76.20% 328 -> 1
* mount: list directories with chunk lists omitted
The two meta cache listings behind a readdir are the only callers, and
neither reads a chunk. On 200k single-chunk files one enumeration goes
from 364ms to 277ms and drops a million allocations.
The read-through listing still fetches whole entries from the filer,
which would need the request to say it wants attributes only.
* mount: give the readdir benchmark's entries a chunk
Chunkless entries made the decode look far cheaper than it is, which is
the part of a listing worth measuring.
* filer: let a listing ask for entries without their chunk lists
The read-through readdir fetches whole entries over gRPC, and for a wide
directory the chunk lists are most of what crosses the wire and most of
what the client then unmarshals. A 4MB-chunked file is 113 bytes of
entry against 46 without its chunk.
ListEntriesRequest gains omit_chunks. The size a client needs is already
in the attributes, where the store decode folded the chunk extents in,
so dropping the list costs the client nothing.
The filer still reads the entries whole. A listing is where a TTL-expired
entry gets collected and deleted, and deleting one needs its chunks to
find the data, so omitting them there would leak. Only the response is
trimmed.
The hint moves to filer_pb so one context flag serves both transports:
the gRPC request sets omit_chunks, and a listing served from the local
store skips building the chunks. Cache population is unaffected either
way, since EnsureVisited starts from its own context.
* filer: reject a chunk the full decoder would reject
The walk skipped a chunk's bytes without looking inside them, so a
FileChunk carrying a corrupt nested fid, or a string that is not valid
UTF-8, sailed past the listing decoder while every other read of the same
entry still failed. The file listed with a plausible size and then gave
EIO on open, and corruption that used to fail the listing loudly was
hidden instead.
The chunk bytes are the one part of the blob the generated unmarshaller
never sees, so the two checks it would have made are made here: a
submessage has to parse, and a proto3 string has to be valid UTF-8.
FileChunk's only submessages are FileIds of scalars, so walking them is a
complete check. A descriptor-driven test fails if FileChunk ever gains a
field of either kind that the walk does not know to check, which is the
part that keeps this honest as filer.proto grows.
Taking the scratch buffer lazily, only once a chunk is actually dropped,
also takes the pool out of the path for entries that have none. Those
were measurably slower than the full decoder before; they are now level
with it. Each chunk's length prefix is parsed once rather than twice.
chunks full attrs-only vs base
0 171.4n 176.9n ~ (p=0.670)
1 366.6n 259.4n -29.24%
4 1.034u 500.2n -51.60%
16 3.905u 1.464u -62.52%
64 13.48u 5.195u -61.46%
* filer: carry the size before dropping chunks over the wire
Dropping the chunk list assumed every store folds the chunk extents into
FileSize when it decodes. A store that keeps entries as JSON rather than
as an encoded Entry never re-derives it, so an object written with a zero
FileSize kept its real size only in the chunks, and stripping them left
the client reading the file as empty. Stamp the size into the attributes
first, which costs nothing and does not depend on how the store loaded
the entry.
* mount: test that the readdir context reaches the store decode
Everything else exercises the decoder directly, so a refactor that
stopped threading the context would have reverted the whole thing with
every test still passing.
The benchmark's chunks also carried a constant legacy FileId, which
BeforeEntrySerialization reparses over Fid on the way in, so all 200k
entries stored one byte-identical chunk rather than the varying fixture
it looked like.
GetVolumeIds ranged over a slice and collected the loop indices, so it reported
0-99 rather than the node's volume ids. Nothing calls it: the disk-level
GetVolumeIds, which ranges over a map and is correct, is what ToDiskInfo and
ToMap use. Its private getVolumes helper went with it, having no other caller.
* perf(weed/topology): preallocate the node's volume concatenation
A node's volumes are gathered per disk and concatenated into a slice grown from
nil, so a server with several disks reallocates and copies its way up. The
writable-volume refresh loop does this for every node every few seconds.
BenchmarkDataNodeGetVolumes/8Disks 322551844 B/op -> 121602326 B/op
* perf(weed/topology): fill one slice across a node's disks
Each disk built its own right-sized copy of its volumes, and the node then
copied all of them again into the combined slice. Appending into the caller's
slice makes it one allocation whatever the disk count, which halves even the
single-disk case.
BenchmarkDataNodeGetVolumes
1Disks 121602326 B/op 2 allocs/op -> 60801314 B/op 1 allocs/op
8Disks 121602326 B/op 9 allocs/op -> 60801024 B/op 1 allocs/op
* perf(weed/topology): preallocate the /dir/status volume list
ToVolumeMap boxes every volume on a node into an []interface{} grown from nil,
so the slice reallocates its way up alongside the boxing. The count is known.
* perf(weed/topology): preallocate the volume id list sent to clients
Every filer, s3 gateway, and mount that connects to the master gets one
VolumeLocation per data node carrying that node's whole volume id list, grown
from nil. The count is known.
The ec ids are left alone: shards of one volume can span disks, so the shard
count is an upper bound on the deduped vid count, not the count itself.
ToDiskInfo builds a protobuf message per volume and per ec shard on the disk,
growing both lists from nil. Every VolumeList call runs it for every disk in
the cluster, and the admin dashboard, the plugin worker, several shell commands
and the s3 gateway's per-minute bucket metrics all call VolumeList. Both counts
are already in hand.
ToTopologyInfo over 550k volumes 202.2 MB -> 184.6 MB
A reconnecting volume server reports every volume it has as new, so newVolumes
grew from nil to one entry per volume, reallocating and copying its way there.
Sizing it to len(actualVolumes) instead would allocate the whole list on every
steady-state heartbeat, where nothing is new.
After the deletion pass everything left on the node is also in this heartbeat,
so the difference is exactly what the node is about to gain: all of them on a
reconnect, none in steady state.
First registration of 550k volumes 1041.7 MB -> 667.4 MB
* mount: readdir enters a child in the inode table only when it takes a reference
Only readdirplus into the kernel takes a reference on the children it
reports, and only that reference brings a FORGET later to take the entry
back out. Every other listing was inserting all its children anyway.
On WinFsp that meant a listing looked each child up, took a reference,
and immediately gave it back, so a walk of a wide directory paid three
write-lock acquisitions per entry to leave the table exactly as it found
it. On a plain kernel readdir nothing gives the entry back at all, so
listing a directory of 200k files grew both maps by 200k entries that
were never reclaimed.
A dirent's inode number is informational either way: the kernel must
LOOKUP before it can use a nodeid, and the WinFsp adapter re-resolves
every operation by path. So report the number and let the mapping be
built when something actually looks the entry up.
* mount: take the readdirplus reference without a second full lookup
The entry has just been resolved a few lines above, so redoing the whole
lookup only rebuilds the child path and walks both maps again to reach a
counter. Bump it directly, falling back to the full lookup if a Forget
removed the entry in between.
* mount: benchmark a readdir over a 200k directory
Drives doReadDirectory against a meta cache holding 200k entries, one
round of 4096 at a time, for the three front ends that behave
differently: a plain kernel readdir, kernel readdirplus, and a WinFsp
listing that gets attributes but never returns a reference. Reports what
each leaves behind in the inode table alongside the usual metrics.
The sink declares TakesLookupRef as an ordinary method rather than
through the interface, so the same file runs unchanged against an older
tree for comparison.
* mount: stamp an inode on the benchmark's entries
The filer stores one on every entry it writes, so a real listing arrives
with an inode and never derives its own. Leaving it zero made every child
in the benchmark fall through to the MD5 in AsInode, work no filer-backed
mount does, and charged it to both sides of the comparison.
NewReplicaPlacementFromByte formatted the byte with fmt.Sprintf and parsed the
result back, allocating a string and a ReplicaPlacement every call. The master
calls it once per volume in every heartbeat, and keeps the pointer for the
lifetime of the volume, so a cluster with 1.6M volume replicas carries 1.6M of
these where a handful of distinct values exist.
The table is a flat pointer-free array, so it costs 6KB of static data and no
heap objects however few placements a cluster actually uses.
A byte only ever decodes to a valid placement, so the table is complete and the
error return stays nil.
BenchmarkSyncDataNodeRegistration/100000Volumes 500601 allocs/op -> 300589 allocs/op
The master decodes a TTL per volume in every heartbeat and keeps it for the
volume's lifetime, so a cluster using TTLs carries one two-byte object per
volume replica where at most 256 counts times 7 units exist. Share them, and
decode the uint32 form directly instead of staging it through a byte slice.
Clusters that set no TTL are unaffected; that path already returned the shared
EMPTY_TTL.
BenchmarkSyncDataNodeRegistration/100000Volumes, volumes carrying a ttl
600600 allocs/op -> 500597 allocs/op
2026-08-07 00:51:52 -07:00
Chris LuGitHubcoderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* perf(weed/topology): log writable-state changes, not every check
ensureCorrectWritables ran its three diagnostics whenever it was asked, so a
volume that had always been read-only re-announced that on every registration.
A volume server reconnecting with 550k read-only volumes made the master format
over a million log lines before it could serve anything, which is exactly when
it is already at its memory peak rebuilding the topology.
removeFromWritable already reports the transition, and only when there is one.
Explain it only then. Dropped the separate 'remove from writable' line, which
said nothing that 'becomes unwritable' does not.
BenchmarkRegisterReadOnlyVolumes, 100k volumes
285791480 B/op 1902069 allocs/op -> 234592088 B/op 1302572 allocs/op
* Update weed/topology/volume_layout.go
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Disk.GetVolumes copies the disk's whole volume map into a fresh slice, and every
caller that walks a node's volumes goes through it: the writable-volume refresh
loop every few seconds, ToTopologyInfo on each VolumeList, telemetry, and node
unregistration. Growing from nil reallocates and copies about twice the final
size each time, which at 100k volumes per disk is 60MB of garbage per call.
BenchmarkSyncDataNodeRegistration/100000Volumes 199670102 B/op -> 137728051 B/op
* perf(weed/topology): keep only volume ids in the heartbeat membership set
The map is used solely to test whether a known volume is still present, but it
copied the whole 152-byte VolumeInfo for every volume in the heartbeat. Presize
it too, since the count is known.
BenchmarkSyncDataNodeRegistration/100000Volumes 199670102 B/op -> 180157310 B/op
* perf(weed/topology): diff a heartbeat without copying the volume map
To find volumes the data node no longer reports, UpdateVolumes copied every
VolumeInfo on the node into a fresh slice, then deleted the missing ones one at
a time. At 100k volumes that is a 15MB copy per heartbeat to usually find
nothing. Scan the disk maps in place instead and return only what was removed.
BenchmarkSyncDataNodeRegistration/100000Volumes 180157310 B/op -> 87806436 B/op
* test(weed/topology): benchmark the per-heartbeat volume sync
A volume server re-sends its entire volume list every VolumePulsePeriod, so
SyncDataNodeRegistration is the master's steady-state per-server cost. Give it
a benchmark so allocation regressions show up.
* perf(weed/topology): preallocate the heartbeat volume conversion slice
The slice grows to one entry per volume on the data node, so at 100k volumes
the doubling copies allocate 60MB of garbage per heartbeat. The final length is
known up front.
BenchmarkSyncDataNodeRegistration/100000Volumes 199670102 B/op -> 137725872 B/op
* helm: mount an enterprise license Secret into every component
Running the enterprise image under this chart meant hand-rolling
extraVolumes and extraVolumeMounts on every component. Missing one is
easy and quiet: a component without the license silently drops to
community mode, and on the admin that surfaces only as Data Recovery and
Point-in-Time Recovery refusing to enable, with the master looking fine.
Add global.seaweedfs.license.existingSecret. The Secret is mounted
read-only into master, volume, filer, s3, sftp, admin, worker and
all-in-one, and SEAWEED_LICENSE points every process at the file rather
than relying on the binary's search paths, which depend on the working
directory.
The mount is a directory, never a subPath: kubelet refreshes Secret
contents in place, but a subPath is resolved once at container start and
never updates, which would break license renewal. There is deliberately
no checksum annotation on the pod template either — that would roll every
pod on renewal, the opposite of what is wanted. Verified on kind: the
renewed file reached a running master ~70s after the Secret was patched,
same pod UID, restartCount 0.
Also documents that only the master re-reads the license on a timer
today; the other components pick a renewal up on their next restart.
* helm: keep master data on a claim by default
The master's -mdir holds its Raft log and snapshots, and with them the
cluster's topology UUID — the identity an enterprise license is issued
against. It defaulted to a hostPath under /ssd, which does not follow a
rescheduled pod: the master came back with an empty data directory, a
freshly generated cluster UUID, and a license that no longer matched.
With the chart's default of a single master replica there is no peer to
recover the identity from either.
Default master.data.type to persistentVolumeClaim, sized 1Gi (Raft state
is small). hostPath stays available for anyone who wants it.
This is breaking for existing releases: volumeClaimTemplates is immutable,
so helm upgrade on a release installed with the old default fails with
"updates to statefulset spec for fields other than ... are forbidden".
Verified on kind, along with both ways out — pinning
master.data.type=hostPath upgrades cleanly, and the documented migration
(stop the master, pre-seed a claim named after the StatefulSet, upgrade)
preserves the cluster UUID. Seeding has to happen while the master is
stopped; copying into a live pod loses the state, since the running
master rewrites its Raft files before the restart.
* helm: mount the license on masters only
The master is what reads the license file: it validates it, enforces the
capacity limit and binds it to the cluster UUID. Mounting the Secret on
volume, filer, s3, sftp, admin and worker put it in six more containers
that never look at it, so drop it there and keep master plus all-in-one,
which runs `weed server -master`.
Two fixes from review while here:
- project only the configured key out of the Secret, so an unrelated
key in the same Secret is not exposed to the container. Verified the
key-scoped projection still updates in place: patching the Secret
reached the running master in ~50s, same pod UID, restartCount 0.
- drop SEAWEED_LICENSE from merged extraEnvironmentVars while a
license Secret is configured. It used to be possible to render the
key twice in one container, with the user's value winning over the
path the chart actually mounts.
CI now pins the scope (master only, all-in-one separately), the
key-scoped projection, readOnly, and that SEAWEED_LICENSE renders once.
* helm: fix the documented master-data migration
The seed pod in the migration never mounted the claim it was supposed to
seed, so following the steps verbatim copied the Raft state onto the
pod's ephemeral filesystem and threw it away with the pod — landing the
reader in exactly the empty-claim, new-cluster-UUID state the section
exists to avoid. Give the pod the volume.
The names were assembled as <release>-seaweedfs-*, which is wrong
whenever the release name already contains the chart name or an override
is set; read the StatefulSet name from the cluster instead and derive the
claim from it. Also scope the procedure to the chart's single-master
default, and create the Secret in the release namespace.
Trims the enterprise prose this section had accumulated: this is the OSS
chart, and the master-data default is a durability fix that stands on its
own.
* helm: quote the projected license key, reserve it on the secret env path
A secretKey that YAML reads as a non-string (123, yes, no) rendered
unquoted into the volume's items, so the projection would not match the
Secret's key. Quote both key and path.
all-in-one renders secretExtraEnvironmentVars itself, outside the merge
helper that already drops SEAWEED_LICENSE, so an entry there could still
render the variable twice. Skip it there too while a license Secret is
configured. The master template has no such block, so this is the only
remaining path.
* helm: correct the license helper comments after scoping to masters
* helm: keep hostPath as the master data default
Defaulting master.data.type to a claim broke every existing release:
volumeClaimTemplates is immutable on a StatefulSet, so helm upgrade
failed with "updates to statefulset spec for fields other than ... are
forbidden" before it changed anything.
Keep hostPath as the default and document the claim as the option to
choose — for a new install, or for an existing one via the migration
already in the README. The chart supported both types all along; only
the default moves back.
Every immutable field of every rendered StatefulSet is now identical to
upstream under default values, so an in-place upgrade cannot trip the
API. Verified on kind: install with the unmodified upstream chart,
upgrade to this branch (ok), upgrade again turning the license Secret on
(ok, volume added in place). A fresh install with
master.data.type=persistentVolumeClaim binds its claim as before.
The whole PR is additive now: nothing renders differently until a value
is set.
* helm: scope the migration's StatefulSet lookup to the release
* helm: trim the comments added by this change
* rust: only compare the .dat tail on v3 volumes
Go's verifyNeedleIntegrity does the "does .dat end exactly at the last
indexed needle" comparison inside its v3 branch -- it rides along with
the v3 append-timestamp read -- so a v1/v2 volume carrying an unindexed
trailing record loads read-write and silent. The Rust check ran it at
every version, so booting the Rust server on a legacy cluster warned on
and quarantined volumes the Go server had been serving happily.
* rust: load a disk's volumes concurrently
Opening a volume is dominated by reading its .idx into the needle map,
and the loader did them one at a time, so a disk holding thousands of
volumes needed thousands of serial index reads before the server came
up. Go's concurrentLoadingVolumes spreads the same work over
max(cores, 10) workers; do the same, keeping the directory pre-pass and
the insert serial so only the open is parallel.
* rust: let a failed volume open fall back to the next candidate
Two collections can name the same volume id on one disk. Deduping the
load queue by id claimed the id for whichever candidate the scan saw
first, so a corrupt one shadowed a good one behind it; the serial loader
this replaced only claimed an id once a volume had actually opened.
Carry every claiming collection per id and try them in scan order until
one loads.
* rust: trim the new comments in the volume loader
* admin: count each chunk once in the dashboard total
The dashboard summed file_count from every node's volume list, so a chunk
was counted once per replica and deleted chunks were never subtracted.
Reuse the collection aggregation, which dedupes replicas and EC shard
holders and nets out tombstones.
* admin: the dashboard card counts chunks, so name it that
Volumes store chunks, and a file is split into one or more of them, so
the 'Total Files' card always read far higher than the number of files in
the filer. Rename it to 'Total Chunks' and say so in the tooltip.
* admin: collections pages count chunks once and say so
The collections list and detail pages summed file_count straight off the
topology, so replicas multiplied the count, tombstones stayed in it, and
the detail page ignored EC volumes entirely. Take the numbers from the
shared collection aggregation and label them chunks.
* admin: dedupe replica chunk counts per volume instead of dividing
Dividing each replica's live count by the copy count truncated a chunk
per odd-sized volume, and reported half the count while a volume's
second replica had not checked in yet. Replicas mirror each other's
needles and deletes, so keep the fullest report per volume id.
* admin: fix the collections CSV export column mapping
The exporter read chunks from the EC-volume cell and shifted size and
disk types with it. Read every column the table actually has.
The filer StatefulSet declares db-schema-config-volume unconditionally, referencing <release>-seaweedfs-db-init-config. The chart never renders that ConfigMap and the README documents it as operator-supplied, so any deployment that has not pre-created it ends up with a pod spec pointing at a non-existent object. No container mounts the volume, so this is currently inert, but it is misleading and would become a pod-start failure if a volumeMount is ever added.
* fix: apply S3 Expiry Metadata
* add test Header X-Seaweedfs-Expires-S3
* resolve comments
* test entry lookup by mtime
* filer: skip s3 expiry stamp on versioned entries
The s3 expiry path skips entries carrying a version id, so stamping one
takes away its expiry rather than moving it onto mtime. Files under
.versions/ are written once, so crtime already tracks their needles.
---------
Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: give a versioned metadata-only copy its own chunks
A self-copy that only rewrites metadata clones the source entry, chunk fids
and all, and writes the clone back. With no versioning that is exactly right:
the clone replaces the entry it came from, so one entry owns the needles the
whole time. Under versioning the clone lands in a new .versions/ file and the
source stays live, and nothing refcounts a plain shared chunk list -- deleting
either version (a NoncurrentVersionExpiration rule, say) frees needles the
other still points at, and the next vacuum makes that permanent. rclone hits
this on every upload, since it stamps mtime with exactly this copy.
Take the metadata-only path only where the write replaces the entry it read:
the bare key of a bucket without versioning. Versioned, suspended, and
versionId-pinned copies fall through to the regular copy path, which gives the
destination its own chunks.
* s3: reencrypt a versioned SSE-KMS key rotation instead of reusing the chunks
A same-object copy that changes the KMS key id hands the source chunks straight
back, on the assumption that the copy overwrites the entry they came from. A
versioned bucket writes a new version beside the source instead, so the two end
up sharing needles that nothing refcounts, and deleting either one frees the
other's data. Reuse the chunks only when the destination really is the source
entry; otherwise fall through to the reencrypt path, which also gives the new
version the key it asked for rather than leaving it on the old one.
* s3: make one predicate decide whether a copy replaces its source
The metadata-only branch and the key-rotation strategy both answer the same
question -- does this copy write back to the entry it read -- so let them share
one predicate instead of pairing a same-destination check with it separately at
each site.
* test(s3): fail the copy regression tests when the vacuum does not run
The helper swallowed a failed or non-200 request to the master, so a vacuum
that never ran turned both chunk-ownership assertions into no-ops: the
tombstoned needles were still readable and the surviving version looked fine
either way. Require the endpoint, the request, and a 200.
* ci(s3): run every versioning test in the regression gate
The gate named the tests it wanted, so a new regression test sat there
uncovered until someone remembered this file -- it fooled me into thinking two
tests added in this PR never ran anywhere, when the comprehensive job had them
all along. Invert it: run everything, and name a test only to keep it out. The
delete job beside this one already works that way, and the suite costs about
two minutes.
Only the pagination stress tests are excluded; they build 1500+ versions, skip
themselves without ENABLE_STRESS_TESTS, and have their own make target.
Go's regexp has no negation, so the pattern is still assembled from a listing,
the way the volume-server integration workflow does it. Note the trailing $$:
make eats a lone trailing $ and takes the anchor with it.
* operation: re-assign chunk upload when replica volume is full
When a volume reaches MaxPossibleVolumeSize, needle writes return
'Volume Size Exceeded' and the fan-out in uploadChunkToHolders fails.
Previously the error propagated immediately, killing the entire chunked
upload even though the master has other writable volumes.
Fix: detect 'Volume Size' errors on the fan-out path, call AssignFunc
again to get a fresh volume, and retry (up to 3 attempts). This avoids
backup failures while a single replica volume is full and waiting for GC.
Also add 'volume size exceeded' to the transient error messages so any
retry path that checks IsTransientError recognises it.
* util: fix transient error pattern for volume size errors
The actual error message is 'Volume Size 34361499680 Exceeded 34359738368'
where the numeric size separates 'Volume Size' from 'Exceeded'. The
previous pattern 'volume size exceeded' would never match. Change to
'volume size' which correctly matches any capacity-full error.
* operation: fix reassignment loop nits
- Case-insensitive volume size match (strings.ToLower)
- Propagate AssignFunc error so caller sees reassignment failure
- Reset JWT fallback before each reassignment to avoid retaining stale auth
* util: stop classifying a full volume as a transient error
A volume at capacity does not become writable on the next attempt, so the
entry only bought a retry loop's worth of sleeping before the same failure.
It also reached four consumers that all retry the same target — the deletion
queue, the replication sink, volume lookups, and Retry/MultiRetry — none of
which reassign, and the widened "volume size" substring swallowed the
replica-receive rejection from WriteNeedleBlob too.
The chunked upload path recovers by asking for a different volume instead.
* operation: reassign a chunk with the shared upload gate
shouldReassignUpload already answers this question for the non-chunked path,
keyed off the status the volume server returned rather than its message text.
Reusing it covers a lost replica peer and an unreachable target as well as a
full volume, and it stops a 4xx from being retried on a second volume that
would reject it identically.
Pulling the single attempt out into uploadChunk keeps the retry loop readable
now that it wraps both the fan-out and the relay path.
* test: cover chunk reassignment across volumes
Pins the three outcomes the gate decides: a full volume moves the chunk to a
fresh assignment, a 4xx stays put, and the loop gives up after
chunkAssignAttempts volumes.
* operation: roll back the fid a reassigned chunk abandons
ReplicatedWrite commits the needle locally before it replicates, so a 5xx can
leave a copy behind on a volume the chunk is about to walk away from. Nothing
will ever reference that fid, and an unreferenced needle is not garbage vacuum
can find — it is dead space until the volume is destroyed.
uploadChunkToHolders rolls back only the holders that reported success, which
misses the one whose write landed but whose response did not, and the relay
path had no rollback at all. Delete from every holder of the abandoned
assignment instead; deleting a needle that never landed is a no-op.
* operation: stop the reassignment loop from multiplying work
retriedUploadData already retries a chunk three times against the same URL, so
wrapping it in three assignments made nine POSTs for one chunk. On the relay
path that inner retry is redundant — the loop retries everything it would, and
on a different volume — so cap it at one attempt per assignment and leave the
budget where it was. The fan-out path keeps its inner retries: absorbing a blip
on one holder beats cancelling the rest and re-uploading the whole chunk.
Nothing bounded any of it by time. weed/s3api passes context.Background() so a
chunk survives client disconnect, which also means no deadline cuts the loop
short, and a chunk goroutine holds one of four buffer slots while it spins.
Break out once another chunk has already failed the object.
* operation: keep the reassignment gate's inputs deterministic
uploadChunkToHolders reported whichever holder error won the channel race.
That was cosmetic while the value was only logged; now it decides whether the
chunk moves to another volume, so a 400 and a 500 arriving in either order
made the retry behavior depend on scheduling. Prefer an error the caller can
act on, and the same failure always retries the same way.
A failed reassignment also overwrote the upload error that prompted it, which
buried a full volume behind whatever the filer happened to say. Keep both in
the chain.
The tests grew a JWT per assignment, since the loop re-derives one and nothing
covered it, and the bound is now spelled out rather than compared against the
constant that defines it.
* operation: roll back the last abandoned fid too
The rollback ran only on the path that goes on to reassign, so the attempt
that exhausts the budget — or stops because another chunk already failed the
object, or because the error is not one a different volume fixes — left its
fid behind. That is the case that matters most: no chunk names it, the caller
gets no fid to clean up, and a 5xx can still mean the needle was committed.
Roll back on every failed attempt instead, before deciding whether to retry.
---------
Co-authored-by: timolow <timolow@users.noreply.github.com>
Co-authored-by: timolow <tim@timolow.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: keep a non-S3 action out of the request-shape resolver
ResolveS3Action reads the request shape before it looks at the base action, so
an iam: or sts: action on a request that happens to carry an S3 query parameter
came back as the S3 action for that parameter. An action that already names its
service is resolved; there is no S3 request shape to read for it.
* iam: authorize the standalone IAM server's actions as IAM, not as S3
The standalone `weed iam` server wrapped its single POST / route in the generic
S3 Auth middleware with ACTION_ADMIN. The route has no {bucket}, so the check
ran with an empty bucket and resolved to a coarse S3 action rather than the IAM
one. The embedded IAM surface checks iam:<Action>; the standalone one was never
updated to match.
Both now go through one authorization function, so they cannot drift apart
again. It also rejects the anonymous identity, which has no user of its own to
run a self-service action against, and reads UserName from the body only, where
the handlers read it from.
The suspended PUT dropped the null delete marker before writing, so a failed
write left the .versions pointer naming a marker that was gone. The read path
heals a dangling pointer by promoting the newest survivor, so a key the caller
had deleted came back serving an older version, and the heal persisted that
pointer. Move the retire into afterCreate via the shared finalize, which also
brings the ownership check the copy and multipart paths already have.
* s3: report the effective ownership when a bucket has none stored
GetBucketOwnershipControls read Seaweed-X-Amz-Ownership straight out of the
bucket entry, so a bucket that never had one written reported an empty
ObjectOwnership. The object write path defaults the same missing attribute to
BucketOwnerEnforced, so the API contradicted the behavior it describes.
Resolve the stored value through one helper both readers share, and let
PutBucketOwnershipControls persist unconditionally so setting the default
value still gives DeleteBucketOwnershipControls something to remove.
* test: cover the bucket ownership controls round trip
Pins the behaviors the ownership default fix depends on: a bucket that never
had ownership controls written reports BucketOwnerEnforced, and putting that
same value on such a bucket still persists it, so the delete that follows has
something to remove. The put-then-delete case gets its own bucket -- run after
an ObjectWriter put, it would pass against an implementation that skips only
the initial write.
The acl workflow already runs this package against a live weed mini, so it
needs no wiring.
Every test bucket is its own collection and each grows 7 volumes, so the
suite asks for 140 while the job caps the volume server at 100. Slots come
back only when the volume server's next full heartbeat tells the master the
deleted collections are gone, and the suite finishes inside one 5s pulse:
the run survives on whichever buckets happened to be dropped before that
single tick. The last run cleared by two slots, this one wedged the final
PutObject with 'No writable volumes and no free volumes left'.
* s3: let a suspended-versioning multipart completion replace the null delete marker
In a versioning-suspended bucket a DELETE writes a null delete marker into the
key's .versions directory. CompleteMultipartUpload then writes the new null
version at the regular path but left that marker in place, so the completion
returned 200 and the object listed while HEAD and GET kept resolving the marker
and answered NoSuchKey. PutObject already handles this; do the same on the
multipart path.
* s3: order the suspended-versioning null cleanup behind the multipart write
Removing the null delete marker before writing left a failed completion having
already published the key's newest real version: the marker was gone, the
pointer still named it, so reads rescanned .versions and promoted the older
version. Do both fixups only once the write commits, pointer first so reads
never see a pointer aimed at a marker that is no longer there, and fail the
completion when the pointer cannot be cleared instead of returning 200 for an
object HEAD and GET still miss - a non-ErrNone finalize keeps the upload
directory, so the caller's retry replays it.
Also cover a pre-suspension real version in the regression test.
* s3: skip the suspended null cleanup when a concurrent write won the key
The completion's .versions fixups are unconditional rewrites of shared state and
the routed path runs off the object write lock, so a DELETE landing between the
multipart write and the cleanup had its own null delete marker erased - leaving
a successfully deleted key readable as an older retained version. Re-read the
object first and leave the cleanup alone unless it is still the one we wrote.
This narrows the window rather than closing it; a compare-and-set pointer flip
is the real answer and wants its own change.
* s3: re-read the completed object from the filer that took the write
The guard compared the object against our upload id through the routed read,
which skips an owner it recently found unreachable and falls back local-first.
A write that just landed on the owner could then read as superseded on another
filer, skipping the cleanup and leaving the key unreadable - the bug this set
out to fix. Read back from the filer the write went to instead.
* s3: trim the suspended-completion comments to the non-obvious why
* s3: lift the suspended null-write finalize into a named helper
The pointer-then-marker ordering is policy shared by every suspended null write,
not something the multipart path should be stating on its own; putSuspendedVersioningObject
and the copy path each restate it today. Give it a home next to the versioned
finalize helpers, and reuse the canonical key normalizer and the existing test
helpers rather than open-coding both.
* s3: retire the null delete marker on a suspended-versioning copy
The suspended CopyObject branch cleared the .versions latest pointer but left the
null delete marker a preceding DELETE wrote. While the regular-path object owns
the null slot that marker is shadowed, so it reads and lists correctly - but it
resurfaces as a phantom delete for a key nobody deleted once that null version
goes away. Route the branch through the shared finalize.
* s3: keep the suspended null cleanup from erasing a concurrent delete
Retiring the marker on the copy path reopened the race the multipart path had
already closed: a DELETE landing between the write and the cleanup lost its own
marker, so a rescan promoted an older version under a deleted key. Move the
ownership check into the shared finalize, keyed on the attribute that identifies
the caller's write, so both paths get it.
* s3 lifecycle: bound the daily-replay subscription at the pass boundary
A pass opens one meta-log subscription and 16 shard drains, then waits
on all of them. Nothing told the subscription where the pass ends, so
the only exit was the fan-out spotting an event past runNow — i.e. some
unrelated write landing under /buckets after the pass started. On a
cluster that goes quiet the reader parks in Recv, every shard drain
starves on an empty channel, and Run never returns. The job sits at
stage "starting" with the executor slot held and no log line, so
expiry stops cluster-wide until someone restarts the worker.
The pass covers (globalStartTsNs, runNow], so say that: UntilNs on the
subscribe request makes the filer end the stream once it has shipped
that range. The reader then closes the event channel on the way out,
which is what unblocks the fan-out and the drains when the stream
finishes on its own rather than by cancellation.
Same fix retires the other silent hang: a reader that failed early
(subscribe error, stream error) also left every drain waiting forever.
* s3 lifecycle: keep a halted shard from starving the shared fan-out
A drain that halts mid-stream (BLOCKED / RETRY_LATER / an RPC error on
dispatch) returns while the fan-out is still routing that shard's
events. After 256 of them the per-shard buffer is full and the fan-out
blocks on the send, so no other shard sees another event. Run's
WaitGroup never drains, and the teardown that would cancel the reader
sits behind that wait — the pass wedges exactly like an idle
subscription did, with one S3 hiccup as the trigger.
Keep discarding the channel after runShard returns. The events are
past this shard's saved cursor and get re-scanned next pass anyway.
* s3 lifecycle: assert the starved shard actually made progress
The fan-out test only checked that Run returned, which a version that
quietly dropped the second shard's events would also satisfy. Assert
the dispatch landed and the cursor moved.
recordingClient gains a per-object outcome map: the two shards dispatch
from separate goroutines, so pinning BLOCKED by call index was a race
waiting to pick the wrong shard.
* s3 lifecycle: fail the pass when the shared subscription dies
Closing the event channel on reader exit is what unblocks the shard
drains, but it also means a subscribe that never opened, or a stream
that broke mid-pass, now ends every drain cleanly. Run logged that at
V(2) and returned the shard result — so a filer failure produced a
green lifecycle job that had processed nothing.
Surface it as the pass error. Cursors still hold what was processed and
tomorrow resumes there; what changes is that the job stops claiming
success.
Cancellation has to stay a non-error — the shell driver's -runtime cap
is a truncated pass, not a failed one — and a canceled gRPC stream
arrives as a status code, not a wrapped context.Canceled, so isCanceled
checks both forms the way the rest of the tree does.
* s3 lifecycle: decide reader cancellation by intent, not status code
A stream we cancel and a stream the filer cancels both arrive as
codes.Canceled, so classifying the reader's exit by its error let a
truncated pass report success whenever the failure happened to carry a
cancellation status.
Intent is knowable exactly, so read that instead: the pass stops on
purpose only when the caller's context ended (the shell driver's
-runtime cap) or the fan-out hit the pass boundary itself. Everything
else is a broken subscription and fails the pass.
TestRun_ServerSideCancelFailsThePass and TestRun_CappedPassIsNotAFailure
are the same codes.Canceled from the reader with opposite verdicts —
the pair only passes because the decision no longer looks at the error.
* s3 lifecycle: time out a subscription that stops delivering
UntilNs ends a healthy stream and gRPC keepalive catches a dead
connection, but neither reaches a filer that keeps answering pings while
its handler has stopped producing. The pass would wait on that forever,
since s3_lifecycle is the one job type with no execution timeout.
Bound the wait for each response at 20 minutes, and opt into the filer's
idle heartbeats so a caught-up stream proves liveness instead of looking
stalled. The default sits above the filer's 15-minute metadata-gap
recovery budget, so a subscriber legitimately parked on a gap is never
mistaken for a stalled one.
Recv is only interruptible by killing the RPC, so it moves to its own
goroutine behind a per-response deadline. The timer covers only the wait
on the filer — dispatch to Events happens outside it, so a slow consumer
can't trip the watchdog.
Approach and the 20-minute figure are from #10577 by way of comparing
the two fixes; the wiring differs because the reader here ends the pass
by closing its event channel rather than cancelling the fan-out.
* s3 lifecycle: trim the comments added by this branch
Keep the non-obvious why, drop the prose restating what the code says.
* s3 lifecycle: snapshot reader intent where the reader stops
Sampling ctx.Err() during teardown reads it after the drains and cursor
saves have run. A reader that failed while the deadline was still live,
on a pass whose teardown then outlives that deadline, was classified as
an intentional stop and reported success.
Sampling earlier in Run is not the fix either: before the shard wait, a
legitimately capped pass has not reached its deadline yet and would be
misclassified the other way. Intent belongs where the reader actually
stops, so the reader goroutine records it next to the error it returns.
Reported by greptile on #10578.
* s3 lifecycle: cover the worker-dispatched pass with nothing due
The e2e suite drives the shell command in 14 of 15 files; the one test
on the real admin->worker path backdates an object, so its own delete
pushes a meta-log event past the pass boundary and ends the pass. The
branch where a pass has nothing to dispatch was never exercised through
the worker.
Cover it, asserting the pass returns on its own: no admin cancellation,
and the executor slot free for the next one.
This is not a regression test for the wedge. A pass used to end when any
write landed past its boundary, and on a shared test cluster something
usually does — the whole suite passes on the unfixed build, verified.
The deterministic guards stay the dailyrun unit tests; this one would
catch a pass that hangs unconditionally.
The command chose destinations by walking its location list in order, so it
neither preferred a node near the source nor spread a burst of copies. Replica
placement and "this node already holds the volume" move into the Accept
predicate; the ranking and the per-pick reservation come from placement.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* s3: delete a directory marker instead of versioning it
The key "dir/" is stored as the filer directory itself, so a delete marker
cannot stand in for it without hiding the children underneath, and its history
has to sit inside the directory it describes, where listings keep meeting it.
Delete it the way an unversioned bucket already does: remove the directory
when nothing is left under it, demote it to a plain directory when children
remain, and drop a history an older build recorded for it.
* s3: stop resolving directory markers through a version history
Nothing records one for them any more, so the lookups that read it are dead
weight - and the one in the listing was a filer round trip per directory
marker returned, which for a bucket that keeps a marker per directory is the
whole listing cost. A listing reads what a directory stands for straight off
the entry it already has; a unit test pins that N markers cost one ListEntries
rather than N+1. The guard that keeps a history left inside a directory by an
older build from surfacing as a key named after it stays.
* s3: do not let deleting "dir/" destroy the object at "dir"
Writing under an existing object turns that object's entry into a directory
while it keeps its data, so the keys "m2" and "m2/" end up sharing one entry.
Stripping the entry to delete "m2/" therefore wiped the object at "m2" - a
different key, and in a versioned bucket one no delete marker records. Leave a
directory holding uploaded data alone; "m2/" does not name it.
* s3: make the directory-marker delete fail closed and take the write lock
The guard that spares a promoted file only fired when the entry read
succeeded, so a transient filer error fell through to the delete and could
destroy the object at "dir" anyway. Fail the request instead, take the object
write lock so the entry cannot change between the check and the delete, and
report a stale history that cannot be removed rather than leaving it to keep
naming the key in ListObjectVersions.
* s3: check If-Match inside the directory-marker delete lock
The lock belongs to the caller: taking it inside the delete nested it under the
batch handler's own lock, and since every lock from a gateway shares one owner
the inner release would have freed it while the outer caller still assumed it
held it. Both callers now own the lock, the single-object path re-checks
If-Match inside it the way the other delete paths do, and a batch delete of a
trailing-slash key in an unversioned bucket goes through the same marker path
instead of the raw delete. A history lookup that fails now fails the delete.
Replica placement rules and "this node already holds the volume" are
constraints the picker has no way to model. The predicate receives the
candidate's rack and data center, because the constraints needing them are
exactly the ones a bare node cannot express.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
weed/shell is the CLI command layer: every file there registers a command in
init(). Importing it for placement drags that whole surface, and its
registration side effects, into callers that run no CLI.
The picker now depends only on master_pb and storage/types, with a node type
of its own -- smaller than the balancer's Node, which also carries the volumes
it holds, and placement never needs those.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* shell: add a reusable target picker for volume moves
Picks the emptiest node near the source: locality first (same rack, then same
data center), emptiest within a tier. Free bytes decide where the cluster
reports them, free slots break the tie.
The pick is spent in the passed topology, so planning several moves from one
snapshot spreads them instead of stacking every one on whichever node started
emptiest.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* shell: order targets on one metric, and reserve the real volume size
Comparing some pairs on free bytes and others on free slots is intransitive, so
the winner depended on sort order. One node too old to report filesystem bytes
now puts every candidate on slots.
Reserving the tier average let a batch of large volumes overcommit a
destination; callers pass what the move actually consumes.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
Both workflows split the suite with ^Test[A-H] / ^Test[I-S] / ^Test[T-Z].
Test names cluster, so shard 2 drew 50 of the 114 grpc tests and 30 of
the 64 http ones, and spent 13m51s against 7m48s and 9m09s for its peers.
Listing the tests and dealing them out one at a time splits them 38/38/38
and 21/22/21, and keeps splitting evenly as tests are added. The pattern
is computed once into the environment rather than repeated in the summary
step, where the two copies had to be kept in agreement by hand.
The four targets ran in a shell loop at ~3m13s each, so the job took
13m29s and gated the whole workflow by itself: everything else finished
within 8m13s. A matrix runs them concurrently, ~3m45s wall clock.
fail-fast is off so one broken target still reports the other three,
instead of one target per push.
* volume: skip directory fsync on Windows
* ci: run the windows jobs for the whole vacuum path
Both windows jobs start the same weed mini cluster, so both exercise the
volume server's vacuum path, but only one of them watched a single file
in it. Cover the compact, reconcile and load files in both.
* volume: report a failed makeupDiff instead of discarding it
The cleanup removes assigned to the same err the makeupDiff failure was
held in, so an aborted compaction returned nil once both removes
succeeded. The master then recorded the vacuum as committed and the
volume reloaded against the discarded generation.
* volume: correct the fsyncDir comments after the windows skip
Both comments described the old shape, where windows fell through to a
sync whose error was swallowed.
* volume: keep the makeupDiff failure ahead of its cleanup errors
A failed remove of .cpd/.cpx outranked the failure that abandoned the
compaction, so the caller saw the cleanup error instead of the cause.
Log it and return the original, matching the Rust do_commit_compact. A
leftover temp file is rolled back by reconcile on the next start.
* iceberg: withhold the S3 endpoint from credential-vending clients
A client that sends X-Iceberg-Access-Delegation: vended-credentials builds
its storage credential out of the LoadTable config and drops the one it was
configured with. We vend no credentials, so the endpoint we advertised left
DuckDB signing nothing: every metadata and data file came back 403, and its
attempt to refresh the empty credential 404ed on stage-created tables.
Answer those clients with no config at all so they keep their own
credentials. Clients that do not ask for delegation still get the endpoint.
* iceberg: mark load responses as varying on the delegation header
The FileIO config in a table or view load response now depends on whether
the client asked for vended credentials, so a cache between us and the
client must key on that header rather than on the URL alone.
* test: cover the DuckDB vended-credentials access pattern
Runs weed mini with -s3.externalUrl, which is what makes the catalog
advertise an endpoint at all, and checks both halves: a plain LoadTable
still gets the endpoint, while one asking for vended credentials never gets
an endpoint without the credentials to sign with. The DuckDB round trip
creates a table from a query and reads it back, which is the flow that
failed with 403 on every data file.
* s3: stop listing a directory marker whose latest version is a delete marker
A directory marker is stored as the filer directory itself, so deleting the
key "dir/" writes its delete marker into dir/.versions while the directory
keeps its mime and stays a key object. Every listing kept reporting the key.
Consult that history before treating the entry as a key, and demote it in
memory when it is delete-marked so live children still hold the prefix.
Also skip the container's own .versions entry while listing inside it: the
suffix match read it as the history of a nested object named "", which
surfaces as a phantom dir/dir key as soon as a live directory version exists.
* s3: a directory marker with version history is not also the latest null version
The directory entry behind the key "dir/" is that key's null version, but
list-object-versions reported it with IsLatest hardcoded true. After a delete
the key came back twice, once as the delete marker and once as a null version,
both claiming to be latest. Read the pointer under the directory instead.
* s3: resolve directory markers through their version history on GET and HEAD
GET and HEAD short-circuit any trailing-slash key straight to the filer
directory, so a directory marker kept answering 200 after its delete marker
was written. Resolve the key from dir/.versions first when the bucket is
versioned: a delete-marked current version answers 404 with
x-amz-delete-marker, a named delete-marker version answers 405, and a key
with no history keeps today's directory-probe behavior untouched.
* s3: re-creating a directory marker retires its delete marker
PutObject on a trailing-slash key never looked at the bucket's versioning
state, so re-creating a marker after a delete left the latest-version pointer
on the delete marker and the key stayed invisible to every versioned read.
Point the key back at the directory entry, which is its null version, and drop
the null version .versions may still hold — the same two steps a suspended
write already takes, now shared.
* s3: fail a directory-marker request whose version history cannot be read
Every lookup of dir/.versions treated any error as "no history", so a filer
hiccup served the directory entry for a key whose current version may be a
delete marker, reported a null version as latest over one, and let a PUT
report success without retiring the delete marker it was meant to retire.
Only a confirmed absence takes the no-history path now.
* s3: cancel the directory-marker probe stream instead of abandoning it
The probe answers off the first entry and returns, leaving the ListEntries
stream open for the life of the parent context. Give it a context of its own.
* test: cover delete-on-close on the windows mount
Windows software creates temporaries with FILE_FLAG_DELETE_ON_CLOSE and
never deletes them explicitly. The conformance suite showed a file
outliving its last handle — an aborted test left its file behind and
every later test hit a name collision — but nothing in this suite asks
for the flag, because os offers no way to.
Skips where the flag is unavailable rather than passing quietly.
* test: fail delete-on-close on a real error instead of skipping
Skipping on any error meant a refused flag looked the same as a platform
that cannot ask for it, so the test could pass by never running. It now
skips only on that one sentinel and reports everything else.
Also stops printing a nil error when the file is still there after its
last handle closed, and checks the closes it was discarding.
* Give volume.merge the needle size the target actually indexes by
needleBlobFromNeedle returned the size Append reports, which is
Size(n.DataSize) - payload bytes only. The .dat header, the needle map and
WriteNeedleBlobRequest.Size all use n.Size, which additionally covers the
flags, name, mime and lastModified fields.
Every needle volume.merge copied therefore landed with a too-small size. The
target indexed it at that length, so every later read failed the header check
in ReadBytes with a size mismatch, and on v3 the fresh AppendAtNs stamp landed
NeedleHeaderSize+DataSize+NeedleChecksumSize into the blob - exactly on the
flags byte - overwriting flags, name size, mime size and the first mime bytes
with the top of a timestamp. Needles came back with flags 0x18, no name, no
mime and a phantom TTL parsed from two arbitrary timestamp bytes; the ones
that decoded as expired 404 and vacuum would drop them. Since merge rebuilds
every replica from the merged copy, no clean replica survives.
Return n.Size, which Append fills in as it serializes, matching what the
normal write path stores via nm.Put.
* Reject needle blobs whose size disagrees with their own header
WriteNeedleBlob trusts the caller's size for two destructive things: it is
what goes into the needle map, and it is where the v3 AppendAtNs stamp is
written inside the caller's buffer. A caller passing the payload-only DataSize
convention corrupts both, and nothing surfaces until the needle is read back -
by which point every replica may already have been rebuilt from it.
Parse the blob's own header and refuse the write when the two disagree.
Mirrored in the Rust volume server.
* mount: mark windows files archived and ignore a zero timestamp
Windows synthesises NORMAL when a file reports no attributes at all, which
is not the same as ARCHIVE and is what create_fileattr_test checks.
Utimens also wrote a zero timestamp through. Windows sends zero for a field
it is not setting, and storing it put 1970 in the atime overlay, which then
overrode the entry's real time — so a file created a moment ago reported an
access time of 1970 whenever the caller asked through an open handle.
Reading the path instead went down a different route and looked right,
which is why a probe of a fresh file showed nothing wrong.
* mount: match the file type by its mask, and only treat the epoch as unset
S_IFDIR is part of the multi-bit type field rather than a flag, so masking
against it alone also matched a symlink, which shares the bit. A regular
file is now identified by the type mask.
Rejecting every timestamp at or below zero also rejected a date genuinely
before 1970. Only the epoch itself is what Windows sends for a field it is
not setting, so that is all that is refused.
create_fileattr goes back on the known-failures list: the archive fix
works and the test simply moves on to ask for READONLY too, which needs
Chflags. Taking it off was premature.
* mount: drop the time overlays when an inode is released
atimeMap and dirMtimeMap are keyed by inode and were only ever trimmed by
a random eviction at capacity. Inodes are derived from the path, so a
delete and recreate hands the same number to a different file, which then
reported the previous file's access time — a file created a moment ago
answering with a time from long before it existed.
Cleared when Forget actually releases the inode, not on every decrement:
a partial forget still has users. Forget now reports that so callers
holding state keyed by the inode know when to drop it.
* ci: keep getfileinfo listed while its access time is unexplained
Two causes have been fixed and neither closed it, so the honest state is
listed-with-a-reason rather than removed in hope.
* mount: drop timestamp overlays while the inode table is locked
Forget released the inode under the table's lock but cleaned up the
atime and dir-mtime overlays after returning from it. Inode numbers are
derived from the path, so a lookup arriving in that window is handed the
same number back and can store a time that the cleanup then deletes.
Run the cleanup at the release point instead, as a callback under the
lock. The directory-cache purge stays deferred until after the unlock,
where it has to be.
Claude-Session: https://claude.ai/code/session_01EgY2QA3iiPtiu6ww3P2EBn
Key the Rust cargo cache on the toolchain that built it
The cache key was rust-<Cargo.lock hash> with a bare rust- restore prefix, so
one seaweed-volume/target survived across runner images. cargo tracks its own
inputs but not the runner's C toolchain, so build-script output for C
dependencies is reused even when the system libc underneath it changed.
That is how the Rust jobs got wedged: the cached aws-lc-sys objects reference
__isoc23_sscanf and __isoc23_strtol, symbols glibc only grew in 2.38, while the
jobs link on ubuntu-22.04 with glibc 2.35. Every job died at
rust-lld: error: undefined symbol: __isoc23_sscanf
>>> referenced by bcm.c in archive libaws_lc_sys-*.rlib
with nothing in the tree to explain it, and no amount of re-running helped
because the poisoned entry was hit every time.
Fold the glibc and rustc versions into the key so a toolchain change misses the
cache and rebuilds instead of producing an unlinkable target/.
* s3: resolve a bucket owner recorded as an identity
The admin UI and weed shell record a bucket's owner as an identity name in
s3-identity-id and never write the account id the S3 API stores alongside it,
so such a bucket looked unowned: its ACL owner fell back to the default admin
account, and under the default BucketOwnerEnforced ownership every object
uploaded to it was stamped with that account instead of the bucket owner.
Resolve the identity to its account when no account id is recorded, in the one
place both the bucket metadata and the bucket config derive the owner from.
* s3: drop the recorded account when the bucket owner is reassigned
Changing the owner of a bucket created through the S3 API left its old account
id behind, and that outranks the identity when the owner is resolved, so the
new owner never took effect for object ownership or the bucket ACL.
levelDbWrite persists the replay watermark when its updateWatermark
argument is true. Put and Delete passed "watermark == 0", which is true
on exactly the writes that carry no checkpoint and false on the batch
boundary that carries one. The two cases were inverted:
recordCount % watermarkBatchSize != 0 -> watermark 0, flag true
-> re-persists a zero on 9999 of every 10000 writes
recordCount % watermarkBatchSize == 0 -> watermark N, flag false
-> drops the only value worth saving
The stored watermark therefore never left 0. Recovery stayed correct,
because replaying .idx from offset 0 is a superset of replaying from N
and replay is idempotent, so this never surfaced as a failure. It only
meant generateLevelDbFile walked the entire index on every rebuild, and
every needle write paid a second leveldb Put to rewrite the same zero.
Pass "watermark != 0" so the boundary write checkpoints and the writes
in between leave the key alone.
Verified on a 25000-needle volume: the stored watermark now reads 20000
instead of 0, and a rebuild replays 5000 entries instead of 25000.
The new test drives a full batch of Puts and a full batch of Deletes to
cover both call sites.
* ci: run each conformance test on its own
Run as one batch with --no-abort, a test that fails part way leaves its
files behind and the next one fails creating them, so the report showed a
cascade of failures that were really one. The whole rdwr and flush group
passes when run alone, and was only ever collateral.
Each test now gets its own directory and its own invocation, which costs
a process start per test and makes the list mean what it says.
* ci: clear a case directory before reusing it
-Force creates the directory but leaves anything already in it, so a
leftover from an interrupted run would defeat the isolation this exists
to provide.
* ci: list the one real failure isolation exposed
With each test on its own, 31 of 32 pass. The exception is rdwr_mmap_test,
which compares mapped bytes against what was written and finds them
different — a genuine data mismatch that only appeared once the test could
run to completion instead of tripping over a previous one's leftovers.
* log buffer: wake the interval loop on shutdown instead of sleeping through it
loopInterval parked in time.Sleep(flushInterval) and only re-checked
IsStopping when it woke, so a buffer shut down early kept both loop
goroutines - and the PreviousBufferCount+1 slabs of BufferSize they
reach - alive for up to a full interval afterwards. Select on shutdownCh
against a ticker instead, and give the loops a WaitGroup so a test can
observe that they exit.
* test: release the filers the server tests build
Every helper here left its filer's meta log buffer running, so each test
pinned PreviousBufferCount+1 buffers of BufferSize for the rest of the
run: ~3.5GB of live heap across the package, which overruns the address
space on linux/386 and kills the 32-bit job with an out-of-memory throw.
Thread the test through the helpers so the buffer is shut down on
cleanup, and shut the subscribe harness's filer down outright - its
deletion loop keeps the whole filer reachable otherwise. That harness
quiesces its flush path first, since Filer.Shutdown closes the store a
flush still in flight would write through.
* mount: report . and .. from windows directories
WinFsp strips the dot entries for the root itself and expects every other
directory to report them, the way a real NTFS enumeration does: its
dirctl test asserts a subdirectory's first two entries are "." and ".."
and that a hundred files enumerate as 102 entries. Dropping them
unconditionally is what fails querydir_test.
The Go test that guarded the old behaviour went with it: os.File.Readdir
filters dot entries itself, so it could never have observed either way.
* mount: give the windows dot entries their directory type
The readdir fills an attribute block only for real children, so "." and
".." arrived with a zeroed one and were reported with mode 0. Windows
refuses to enumerate a directory whose first entry is not marked as a
directory, which is the assertion querydir_test fails on with
STATUS_OBJECT_NAME_NOT_FOUND.
They now carry the type the readdir already knew. The explorer walk also
names any unexpected entry rather than only counting, so a dot entry
leaking through reads differently from a missing file.
* ci: run WinFsp's conformance suite against the windows mount
The FUSE mount is held to pjdfstest with an empty known-failures list;
the Windows mount had 24 hand-written tests. winfsp-tests is what WinFsp
uses to check a filesystem behaves like NTFS, and --fuse-external points
it at ours instead of the bundled memfs, so it is the same bar in the
same shape: anything failing that is not listed is a regression.
It reaches oplocks, security descriptors, POSIX unlink-and-rename and
directory-buffer resumption — the places a Windows filesystem actually
breaks, and none of which the current suite touches.
known_failures.txt starts with the four groups that cannot pass by
construction. The first run will show what else needs listing.
* ci: make the conformance runner fail loudly instead of running empty
The first run reported "0 excluded entries" and then died with
STATUS_DLL_NOT_FOUND, so it never tested anything while looking like a
normal failing run.
winfsp-tests links against winfsp-x64.dll, which the installer puts
somewhere the loader does not search, so the WinFsp bin directory goes on
PATH. A missing or empty known-failures list is now an error rather than
a silent run with nothing excluded, which would read as a clean sweep
with no known failures. ${env:ProgramFiles(x86)} needs the braces, and a
mount point without a trailing separator makes Join-Path build a path
relative to the drive's current directory rather than its root.
* ci: read winfsp-tests failures from its report, and list the real ones
The first run exited zero with 30 of 50 tests reporting KO, and the job
went green: --no-abort keeps the suite going past a failure and the exit
code stops reflecting them, so trusting it meant the check could not fail.
The report is now parsed for KO lines and each one named in the error.
known_failures.txt is populated from that run rather than guessed. The
groups are real gaps, not suite quirks: cached and overlapped IO fails as
a block, delete-while-open has no pending state, Windows file attributes
and creation time are not round-tripped, and directory enumeration does
not resume from a marker.
* ci: stop excluding the extended attribute tests
Forwarding landed, so the group runs instead of being taken on trust —
which is the only coverage it has had.
* mount: tell windows about changes made elsewhere
Nothing invalidates a Windows client's cache from this side, so a file
created or removed by another mount, the S3 gateway or the filer API
stayed invisible in Explorer until the user refreshed by hand. The mount
already receives those events; they just had nowhere to go.
WFS gains a listener for every applied metadata event, and on Windows
that turns into the WinFsp notification for the path. A rename reports
both ends, since the destination's own event may never arrive when it
falls outside this mount.
* mount: report a removed directory as a directory
Entry is nil once a path is vacated, so asking it whether the thing that
went away was a directory always answered no and every removal was
reported as a file. Windows watches the two through different filters, so
a folder removed elsewhere never refreshed.
The invalidation now carries what used to be there, which the event
already knew and simply was not passing on.
* mount: report a rename destination once
The event stream already carries a second invalidation describing the new
path, so reporting RenamedTo here sent the destination twice — and always
as a create, so a moved directory arrived as a create followed by a
mkdir.
weed/mount implements all four xattr operations and the filer stores the
values, but the Windows adapter overrode none of them, so cgofuse's
defaults answered every call with 'not implemented'. WinFsp advertises
extended attribute support either way, because cgofuse registers the
callbacks unconditionally, so applications were told the volume has them
and then refused on every use. Attributes written from Linux were
invisible from Windows.
Untested in CI: exercising Windows extended attributes needs the native
NtSetEaFile path rather than anything in os or PowerShell.
* mount: add the WinFsp filesystem adapter
WinFsp speaks a path-based FUSE dialect; weed/mount implements the
inode-based raw protocol the Linux kernel uses. This translates between
them so Windows runs the same filesystem code as everywhere else rather
than a second implementation: paths resolve to inodes one Lookup at a
time, and the raw operations run unchanged underneath.
Errno translation is spelled out rather than passed through. Go numbers
Windows errnos as offsets from APPLICATION_ERROR, so the raw value would
mean something unrelated by the time WinFsp read it.
Hard links return ENOSYS since WinFsp has none, and byte-range locks stay
with its kernel driver rather than the mount's lock table.
Not reachable from the mount command yet.
* mount: build the winfsp errno table with explicit precedence
Platforms alias errnos differently: freebsd has no ENODATA and linux makes
ENOATTR the same value as it. A map literal with colliding constant keys
does not compile, so build the table and let the first entry win, keeping
the general codes their own meaning.
* mount: wire the winfsp adapter into the mount command
RunMount was one function doing filer setup, mount-point preparation and
serving. The setup is the same everywhere, so it moves to mount_common.go
and each platform keeps only what differs.
Windows differs mostly in the mount point: WinFsp wants a drive letter or
a path that does not exist yet, so none of the unix preparation applies,
and a bad one is worth rejecting up front because WinFsp reports failure
as a bare false. Adds -windows.caseInsensitive for software that expects
Windows naming rules.
* ci: mount on windows and exercise it
Builds weed.exe, installs WinFsp, starts a cluster, mounts S: and runs a
test suite against it: round trips at several sizes, offset writes,
rename, delete, nested directories, concurrent writers, and a directory
wide enough to stand in for the case that prompted this.
Nothing else here can run the Windows mount, so without this the adapter
is only known to compile.
* ci: build the windows mount without cgo
The runner has MinGW, so cgo is on by default and cgofuse compiles its
cgo variant, which needs WinFsp's headers. The nocgo variant loads the
DLL at run time and is what the released weed.exe uses.
* mount: make the winfsp path splitting portable and test it
resolve and resolveParent had the splitting inline in a windows-tagged
file, so the cases that matter most there — both separators, empty and
dot components, the root having no parent to create in — could not be
tested on any runner that builds this.
* test: check the windows mount persists across a remount
Reading a file back through the same live mount proves nothing about
durability; the answer can come from the mount's own caches. Write the
fixtures, confirm the filer serves them with the mount out of the path,
then re-read after a teardown and remount.
* test: cover the windows mount operations that had none
Truncate, append, chtimes and the hard-link refusal were implemented but
never exercised, and the errno table was only unit-tested for mapping,
never end to end. Adds names that have to survive the UTF-16 boundary,
rename over an existing target and across directories, and concurrent
handles on one file rather than one file each.
* ci: dial the filer over ipv4 and run the persistence phases
localhost resolves to ::1 first on windows and the cluster binds ipv4
only, so the mount's grpc dial was refused while the http readiness
probe passed by falling back to ipv4.
* ci: pin the cluster to loopback and probe ports by connecting
weed mini advertises the runner's LAN address and binds filer grpc there,
so the mount's dial to 127.0.0.1:18888 was refused while http answered.
The readiness probe also passed with nothing on 18888: Test-NetConnection
reported success for a port that then refused a connection, so it now
opens a socket instead.
* ci: report listening ports before mounting
The readiness probe connects to the filer grpc port and the mount is then
refused on it, which cannot both be true; print the actual state.
* ci: run the cluster, mount and tests in one step
The runner tears down a step's process tree when its shell exits, so the
cluster started in an earlier step was already gone: the readiness probe
passed against a live filer, the step ended, and the mount then found
nothing listening. A diagnostic step reported no weed.exe at all.
Everything that needs those processes alive now shares a step.
* mount: key windows file io on the handle, not the path
Read and Write walked the path on every call to fill in a NodeId the raw
filesystem never reads: both look the file up by handle. Under eight
writers creating files in one directory the walk transiently missed and
the write failed with ENOENT before reaching the filesystem at all.
Same for flush, fsync and the release calls. O_EXCL now fails on an
existing name instead of taking it over, and Symlink is refused: the
entry is easy to create but WinFsp only follows it once the reparse
point is wired up, so it read back as an empty file.
* mount: translate cgofuse open flags for windows
cgofuse reports MSVC's numbering and the raw filesystem tests Go's, so
only the access mode and O_TRUNC lined up: O_EXCL arrived as O_APPEND and
O_CREAT as nothing at all.
Also report which handle a failed write was using, to tell a handle that
was never issued from one released while still in use.
* mount: report which step of a windows create failed
A concurrent create fails with ENOENT and the path walk, the parent
lookup and the create itself are indistinguishable from the caller.
* ci: send weed logs to stderr on windows
glog writes to its own files by default, so the mount's own error output
never reached the redirected log. Its flags are global and have to come
before the subcommand.
* mount: resolve known paths from the inode table on windows
Every create walked the parent chain with a filer lookup per component.
With eight writers creating files in one directory that is hundreds of
concurrent lookups of the same parent, and lookupEntry reports an
authoritative ENOENT when the directory is cached, the entry is not in
the cache and the inode table has no record — a window a concurrent
refresh can open for a directory that plainly exists.
A path the mount already tracks now resolves straight out of that table.
* test: sync the windows persistence fixtures before closing
The mount is killed rather than unmounted, so anything still queued for
flush is legitimately lost and the test was measuring crash durability
while calling it persistence. A 9MB file lost four chunks that way.
* mount: keep the lookup refresh on the target path
Resolving a tracked path straight from the inode table skipped Lookup,
which is also what refreshes the entry: a truncate then read back the
pre-truncate size. Only the parent chain takes the shortcut now, which
is where the concurrent creates were racing anyway.
* mount: log every windows resolve failure
Open suppressed ENOENT and Getattr logged nothing, which hid the two
callbacks that can report a missing file during a create.
* mount: drop dot entries from windows directory listings
readdir reports "." and ".." for the kernel, but Windows enumerates a
directory without them and displays whatever it is handed, so a folder of
200 files listed 202. Go's ReadDir filters them, which is why only the
PowerShell walk caught it.
* mount: flush queued writes when windows mount is interrupted
The signal handler exits the process the moment its hooks return, so the
WaitForAsyncFlush after Serve never ran on ctrl-c and queued writes were
dropped.
* mount: let windows mount over an empty directory
WinFsp turns a directory mount point into a reparse point, which NTFS
allows on an empty directory and refuses on a populated one. The check
rejected every existing directory, so the ordinary habit of creating the
mount point first failed with a message saying it should not exist.
CI now mounts over a pre-created directory and writes through it.
* ci: run the windows mount check on any pull request
It is the only thing that exercises the Windows mount, so restricting it
to pull requests based on master skipped it for stacked ones. Replaces
the branch name that was pushed to trigger it.
* mount: do not log a missing windows entry as an error
Windows probes for entries that do not exist as a matter of course, so
ENOENT from getattr and open is an answer rather than a fault and would
have filled the log.
* mount: take the fast path for parent chains in every windows resolve
Narrowing it to resolveParent left Getattr and Open re-walking the parent
with a filer lookup per component, and those are what Windows calls
before a create: eight writers in one directory still raced a meta cache
refresh there. Only the final component needs the Lookup refresh.
The pass that suggested otherwise came from a run five times slower than
the failing ones, where the race had no room to appear.
* mount: drop the windows path resolution shortcut
Resolving from the inode table skipped the Lookup that refreshes an
entry, and a truncate then read back its old size. Applying it only to
the parent chain kept truncate correct but left concurrent creates
failing, and applying it to the final component too inverted that. The
two cannot both be satisfied this way, so this returns to looking up
every component and leaves the concurrent create failure open.
* mount: fall back to the open handle when a deferred entry is evicted
A create that defers the filer write leaves the entry only in the local
cache. Creating many files at once pushes the directory past the hot
threshold and evicts it, taking that placeholder with it, so a lookup
went to the filer, found nothing, and reported a file that plainly
exists as missing.
The handle still holding the unflushed entry is authoritative for it.
Caught by concurrent creates over a Windows mount, which resolves a path
on every call rather than relying on a kernel dentry cache.
* mount: let cgofuse resolve to the version the module graph requires
rclone already depends on cgofuse at a newer commit than the v1.6.0 pin,
so readonly builds refused the go.mod until it matched what MVS picks.
The interface and flag values the adapter uses are unchanged there.
* mount: wait for a pending async flush before looking up on the filer
Open, unlink and rename already wait, but a plain lookup went straight
to the filer and read pre-close metadata: truncate a file, close it, and
a path probe during the flush window reported the old size. The kernel
attr cache hides this on linux; a front end that resolves paths on every
operation hit it directly.
* mount: reject a umask wider than the file mode it becomes
ParseUint allowed 64 bits and the result is narrowed to os.FileMode,
which is 32, so an out-of-range umask truncated silently instead of
being reported as unparseable.
* mount: address review findings on the windows mount
WaitForAsyncFlush closed its channel unconditionally and shutdown reaches
it from both the interrupt hook and the path that resumes after serving,
so a ctrl-c could panic on a second close.
The deferred-entry fallback read an open handle's entry without its lock,
which is what the other two readers of that field take so FromPbEntry
does not walk the chunk slice mid-append. The async-flush wait also sat
ahead of the meta cache, making every stat of a recently closed file
queue behind uploads; it belongs just before the filer is consulted.
Windows entries were persisted as uid 0: the raw filesystem stores
InHeader's owner and the adapter left it zero. They now carry the
identity the mount was started with.
The errno table used Linux numbering while cgofuse decodes MSVC's, so
ENAMETOOLONG arrived as EDEADLK and five others were likewise wrong; a
windows test pins each value to cgofuse's own constant.
Also: break the filer handshake loop on success rather than always
running ten rounds, accept a drive letter written S:\\, report a missing
WinFsp instead of panicking, keep commas out of the volume label, and
drop -windows.caseInsensitive, which told WinFsp the mount folds case
while lookups stayed exact.
* mount: return windows lookup references so the inode table stays bounded
Every operation that hands back an EntryOut grants a reference the Linux
kernel returns with FORGET. WinFsp has no FORGET, so the adapter took one
per path component per call, plus one per child of every readdirplus, and
never gave any back: inodeToPath grew for the life of the mount. Walking
the 200k-file directory this exists for stranded 200k references.
The adapter now plays the part the kernel plays. Each resolution releases
what it took, and an open handle keeps the reference for its inode until
Release, counted because the raw filesystem reuses one handle for repeated
opens. Holding it is not optional: completeAsyncFlush skips the metadata
flush when the saved path no longer maps to the inode, so releasing early
would lose a close's metadata.
Also stops persisting the display owner. -o uid=-1 makes WinFsp report the
calling user whatever we say, but the value handed to the raw filesystem is
written to the filer, and 4294967295 is what every other client would read.
-windows.uid and -windows.gid set what is recorded.
* mount: fix windows behaviours the reference implementations guard against
WinFsp has no ro option — it discards the flag and leaves the volume
writable — so -readOnly accepted writes and deletes. The refusal now
happens in the operations themselves.
Windows sends times around its own 1601 epoch, which arrive as a large
negative second count; casting them through stored a year-1601 timestamp
that every other client then read. Those are now left alone. rclone
carries the same guard.
Chown returned ENOSYS, and WinFsp passes a chown failure straight out of
SetSecurity, so Explorer's Security tab and icacls failed for edits that
were not about ownership. It now accepts and discards.
Only create and mkdir presented a caller; the rest sent uid 0, which
hasAccess treats as root, so deletes and renames skipped the permission
check that creates got. Every operation presents the same identity now.
A drive letter written S:\ reached WinFsp unnormalised, which recognises
a drive only as exactly two characters and then failed as a directory
path. A test also pins the open flag translation, since swapping O_EXCL
and O_TRUNC would turn 'fail if it exists' into 'truncate it'.
* mount: answer windows getattr and truncate from the open handle
WinFsp keeps the path a handle was opened with and never updates it when
the file is renamed, so resolving the path again fails on a handle that is
still perfectly valid — the ordinary write-temp-then-rename save pattern.
The handle already knows its inode, which also removes a full path walk
from two operations WinFsp calls constantly.
Readlink on the root now refuses. WinFsp probes there to decide whether
the volume has symlinks and enables them unless it fails, and with them on
it resolves a path a component at a time, each one reaching us as its own
walk — all for a feature Symlink already refuses.
* mount: require the windows mount directory not to exist
WinFsp creates the directory itself with FILE_CREATE and removes it when
the filesystem goes away, so an existing one — empty or not — fails with
"mount point in use". Allowing an empty directory was wrong, and the CI
check that appeared to prove otherwise was the vacuous one: listing a
plain directory succeeds whether or not anything is mounted on it, so the
step passed while the mount had failed and the writes went to local disk.
That check now waits for the reparse point, which is what caught this.
* mount: apply review comments on the windows mount
-windows.uid and -windows.gid reached the adapter but not the filesystem
parameters, which is what carries the owner written to the filer, so the
flags changed nothing.
Readdir re-resolved the path while Getattr and Truncate answer from the
handle; a directory renamed during an enumeration then failed on the
stale path WinFsp still holds.
Utimens now honours UTIME_OMIT instead of writing whatever came with it.
* mount: tag the unix-only lock tests away from windows
The production lock files were tagged when the package was made to build
on windows, but the tests that exercise them were not, so anything that
compiles tests for windows still failed on syscall.F_WRLCK.
* ci: vet the mount tests for each target too
Only compiling the non-test build let an untagged test keep a per-OS
syscall constant without anything noticing.
* mount: keep the xattr flag constants off freebsd
x/sys/unix has no XATTR_CREATE or XATTR_REPLACE there, and weedfs_xattr.go
is already tagged away from freebsd for that reason. Putting them in a
!windows file dragged them back in, so master stopped building for
freebsd.
* ci: cross-compile freebsd and darwin too
The windows-only check missed a freebsd break in the very file it was
added to guard, because nothing else on a pull request compiles them.
* iceberg: accept bare bucket names and ARNs as the catalog warehouse
Only s3://<bucket>/ was recognized. A warehouse spelled as a bare table
bucket name or as the s3tables bucket ARN -- the two forms users reach for
first, the latter being what AWS S3 Tables itself takes -- was silently
dropped, so every call landed on the default "warehouse" bucket and failed
with "table bucket warehouse not found".
* iceberg: report a missing table bucket as 404, not 500
Pointing a client at a table bucket that does not exist -- which every
client with no warehouse set does, since the default bucket "warehouse"
rarely exists -- returned InternalServerError with a message naming a
bucket the client never asked for. Answer 404 and say how to select one.
* admin: show the warehouse in the PyIceberg example
The example connected without one, so it always resolved to the default
table bucket and every client that copied it failed on the first call.
* test: pin bearer auth against a table bucket that exists
The subtest called the catalog with no warehouse and accepted 500 as proof
that auth had passed, since the default bucket does not exist. A missing
table bucket now answers 404, which the test read as an auth failure. Give
it a real table bucket so only 200 passes.
* test: assert the missing-bucket guidance reaches the client
The status and error type were checked but not the message, which is the
part of the mapping that tells a user how to select a table bucket.
* test: encode the warehouse query value
The ARN case pasted raw colons and slashes into the query string. Go's
parser tolerates them, so the test passed without modelling how a client
actually sends the request.
The server module has its own go.mod and replaces the root module from
../.., so bumping prometheus/client_golang in the root leaves this one
pinned below what the replacement needs and 'go build' refuses to run.
* telemetry: stack volume servers per cluster over time
The fleet-wide server count says how many volume servers reported, but not
who they belong to. Carry per-cluster counts in /api/cluster-sizes and draw
them the same way as cluster sizes, sharing one cluster ranking so a cluster
keeps its colour across both stacks.
* telemetry: drop the total disk usage chart from the dashboard
The stacked cluster sizes chart right below it has the same fleet total as
its stack height, plus the per-cluster breakdown. /api/metrics still serves
the aggregate for anyone graphing it elsewhere.
* s3: register an identity's inline account instead of collapsing it into admin
Credential stores persist an account inline on the identity and never
emit a top-level accounts list, so every user created through the IAM
API or the admin UI with an email hit the "non exist account ID" branch
and was given the shared admin account. Distinct users then presented
the same owner id, so ownership checks could not tell them apart and
each passed for the others' buckets.
Treat an id missing from the account map as undeclared rather than
invalid: register it, keeping an email another account already claimed.
Both load paths now resolve the account through one helper.
* s3: refresh an undeclared account from the identity that carries it
The merge path starts from the live account cache, so an identity
upserted with the same account id but a new email or display name kept
the cached copy: the new address never reached the email index and the
replaced one still resolved. Changing a user's email through the admin
UI takes exactly that path.
An account registered from an inline block is only described by the
identity carrying it, so refresh it and move its email claim. Accounts
from a top-level list and the predefined defaults are marked declared
and stay authoritative.
* s3: let an account reclaim an email once its holder moves away
Two identities can carry the same email, and the second to load leaves
the lookup with the first. Returning early when the incoming metadata
matches the cached account meant the loser never re-ran the claim, so an
address freed by the holder's update resolved to nobody until the loser
itself changed. Re-index on the unchanged path, which is a no-op while
another account still holds the address.
TestSubscribeLoop_FlushProvenGapSkipsToRetained deleted the log files
once the eviction watermark's own flush had landed, while the windows
sealed after it were still queued. Those flushes then wrote their files
back, and the subscriber served them from disk instead of taking the
gap-skip path, so the windows whose files really were gone came out
missing. Wait through the last sealed window instead.
* mount: drop the unused go-fuse fs package dependency
WFS embedded fs.Inode but never used any of its methods, and the only
other reference was RENAME_EXCHANGE, a constant sitting next to three
literals. Removing both drops fs and five internal packages from the
mount build graph.
* mount: build the package on windows
Windows has no fcntl lock types, no O_ACCMODE and no x/sys/unix, so a
handful of constants kept weed/mount pinned to unix even though the code
using them is portable in-memory logic. Route them through per-OS shims
and give setBlksize a windows no-op.
The POSIX lock table now compiles on windows but stays unreachable:
WinFsp resolves byte-range locks in its own kernel driver, so nothing
will feed it there.
go.mod points at a go-fuse branch commit and needs repinning to a release
tag once that lands.
* ci: cross-compile for windows
Nothing caught the unix-only constants creeping into weed/mount until a
release build failed.
* mount: let readdir feed a sink instead of the kernel buffer
doReadDirectory wrote directly into fuse.DirEntryList, which is the
kernel's wire format. A front end that is not the kernel would have to
pack entries only to parse them straight back out.
Route it through DirEntrySink instead. ReadDir and ReadDirPlus pass the
reply buffer, so nothing changes for the FUSE server.
* mount: pin go-fuse v2.9.4 for the windows build
WFS embedded fs.Inode but never used any of its methods, and the only
other reference was RENAME_EXCHANGE, a constant sitting next to three
literals. Removing both drops fs and five internal packages from the
mount build graph.
* fix(filer): don't skip unflushed events on metadata subscription gaps
A subscriber that falls behind the in-memory log ring during a write
burst could have its read position jumped past events that were evicted
but not yet flushed to disk — silently lost for filer.backup/filer.sync/
mount subscribers. Route all three gap-skip sites through
resolveDiskGapResume: only skip past windows older than a settled
horizon (2*LogFlushInterval); recent gaps wait for the flush and
re-read disk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer): harden metadata gap-skip guard
Address review findings on the settled-horizon guard:
- local subscriptions: gate the skip on the buffer's flush watermark
observed before the disk read (resolveLocalGapResume) — a disk miss
is then proof the gap is empty, with no wall-clock assumptions
- aggregated subscriptions: cap skips strictly below the horizon
boundary (persisted reads exclude ts <= cursor) and pace capped
advances, so the sliding horizon cannot cause disk-probe spinning
- replace unbounded sync.Cond waits with a bounded select on the
buffer's subscriber channel + retry timer + ctx cancellation,
eliminating the lost-wakeup stall
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer): close remaining gap-skip loss paths from re-review
- log_buffer: a sentinel-offset (time-based) read below the earliest
in-memory entry silently started at earliest, skipping a window that
may hold evicted-but-unflushed events. Track the ring's eviction
watermark (lastEvictedTsNs) and keep the inclusive fast path only
when nothing at/after the position was ever evicted; otherwise
return ResumeFromDiskError so the subscription gap guard decides.
- capped horizon advances stay on the disk-probe path (never expose a
mid-gap position to the memory read) and keep pacing
- gap jumps land just below earliest: positions are exclusive, so the
earliest entry itself is still delivered
- subscriber notification keys include clientId/epoch so a replacement
stream never inherits a channel the old stream's cleanup closes
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(log_buffer): generalize the eviction-watermark read gate
Third-review findings:
- epoch/zero-time reads bypassed the eviction watermark: gate them the
same way, so a SinceNs=0 subscriber cannot silently start at the
earliest retained entry after an unflushed window was evicted
- apply the watermark gate regardless of the cursor's batch offset
(batch offsets carry no meaning for time-based reads); this also
serves adjacent cursors (earliest == current+1) from memory instead
of stalling them in the gap loop
- LoopProcessLogData reader names include clientId/epoch, since they
are registered as subscriber keys internally (same collision as the
outer notification keys)
- test: pin the gate (below/at watermark, epoch-after-eviction) and
update the slow-consumer test to the sharper contract — complete
in-memory history is served from memory; disk only once evicted
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(log_buffer): watermark equality is unsafe for inclusive sentinel cursors
Sentinel (Offset <= 0) time-based cursors search from ts-1ns, i.e. they
read inclusively of their own timestamp — and the evicted window may end
exactly at that timestamp. Allow watermark equality only for exclusive
(positive-offset) cursors; sentinel cursors must be strictly above it.
Also shut down the test buffer and pin the equality cases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer): wait on an empty aggregated buffer instead of falling through
When a disk read finds nothing for a ResumeFromDiskError gap and the
aggregated buffer has no readable entries (zero earliest time),
resolveDiskGapResume declines to advance and control fell through to the
in-memory read — which returns ResumeFromDiskError again immediately,
spinning through the probe cycle without any wait. Fold the case into
the existing recent-gap branch so it waits (notification, cancellation,
or the retry interval) before re-probing, matching the local
subscription path, which already waits unconditionally.
* fix(filer): serve the exact eviction-boundary entry without another flush wait
When the flush watermark has passed the earliest in-memory entry but that
entry sits exactly one nanosecond above the cursor, the exclusive resume
target collapses onto the cursor and resolveLocalGapResume declines to
advance — while a sentinel cursor at the eviction watermark keeps
deferring the in-memory read to disk. Progress then depends on the next
flush cycle. Re-arm the cursor with a positive (exclusive) offset instead:
ReadFromBuffer explicitly allows positive-offset cursors at the watermark,
so the boundary entry is served from memory immediately.
Also promote the aggregated path's horizon-capped gap skip to a warning:
that skip may pass events a stalled flush lands later (the aggregated
ring has no flush watermark to gate on), so operators should see when a
flush stall outlasts the settled horizon.
* fix(filer): resume a gap skip with an exclusive cursor
The resume position landed one nanosecond below the earliest in-memory
entry but kept the inclusive sentinel offset. When that timestamp is also
the eviction watermark, the read gate answers ResumeFromDiskError for an
inclusive cursor, the disk read finds nothing, and the resume target
collapses onto the cursor, so neither helper can advance: the subscriber
parks on timed waits forever.
A skip is only taken once the gap is proven empty, so nothing remains to
deliver at the resume timestamp. Resume exclusively instead, and let an
inclusive cursor already sitting on the target count as progress.
* fix(filer): gate aggregated gap skips on eviction, not wall clock
Wall-clock age never proved persistence. The aggregated ring has no flush
watermark because peers persist their own local logs, so a horizon of
2 * LogFlushInterval was standing in for one. While the disk stayed empty
the horizon kept sliding forward and walked the cursor past
evicted-but-unflushed events one window at a time - the volume-outage
stall this change set exists to survive.
The ring does carry a real proof: its eviction watermark. If nothing at or
after the cursor was ever dropped, memory still holds every entry after it
and the gap is provably empty. Below the watermark entries were dropped
and only the producing peer's flush can supply them, so wait for that
flush instead of advancing. The skip that remains is the one the ring can
prove, which keeps the infinite-loop guard for a genuinely empty gap.
* perf(filer): stop re-probing disk on every metadata append
A subscriber parked on a gap woke on the log buffer's subscriber channel,
which fires on every append. On the aggregated path each wake re-ran a
full ReadPersistedLogBuffer - a ListDirectoryEntries plus a readahead
goroutine - so one parked subscriber turned every cluster-wide metadata
event into a store query, exactly while the cluster is already struggling
with the flush stall that parked it. An append also cannot settle the gap:
what this waits on is a peer persisting its own log, which nothing local
signals. Wait on the timer alone there.
The local buffer does signal its flush on that channel, so keep it, but
drain the stale token first: otherwise the appends riding the same channel
spin the wait at write rate. The retry interval covers the notification
the drain discards.
* fix(filer): surface a metadata subscriber parked on a gap
Refusing to skip an unresolved gap trades silent loss for a silent stall,
and a stall is no easier to diagnose: filer.sync and mount followers just
stop advancing, with no error on either end. The only trace was a V(3)
line nobody runs with.
Warn on entry and once a minute after, and carry a subscribe_gap_stalled
gauge for the duration, so a flush that never lands shows up as a stalled
subscriber rather than a consumer that mysteriously went quiet.
* refactor(filer): drop the metadata listener cond with no waiters left
Both listenersCond.Wait() sites are gone, so listenersWaits never leaves
zero, the guard in the filer's notify callback never fires, and the two
Broadcast calls around client registration wake nobody. Remove the cond,
its counter and its lock, and pass a nil notify func: the log buffer
already skips a nil one, and the subscriber channels carry these wakeups
now.
* test(log_buffer): drive the eviction watermark through a real seal
Both eviction tests wrote lastEvictedTsNs directly, leaving the one line
that sets it uncovered: copyToFlushInternal has to read slot 0 before
SealBuffer shifts it out, and moving that read one statement later still
passed every test. Append past the ring instead, and check the watermark
appears only on the seal that drops a window, matches that window's stop,
and then advances. The buffer under test never flushes, matching the
aggregated meta ring where the watermark is the only emptiness proof.
* refactor(filer): build the subscriber reader name once per stream
It was rebuilt on every loop iteration from values that cannot change for
the life of the stream. Hoist it next to the notification key it mirrors.
* docs(filer): tighten the gap-handling comments
Several ran to eight lines restating the same reasoning at each site.
Keep the non-obvious why, drop the retelling.
* fix(filer): mark a confirmed disk position as an exclusive cursor
A disk read returns the timestamp of the last entry it handed to the
subscriber, but the cursor built from it stayed inclusive. Land that
cursor exactly on the eviction watermark and the read gate sends it back
to disk for an entry the disk just delivered; the re-read finds nothing,
neither emptiness proof holds for an inclusive cursor there, and the
subscriber parks - until some later flush, or forever while one stays
stalled. Everything after the watermark was sitting in memory the whole
time.
Carry the offset instead, so it also covers the ring evicting onto the
cursor after the read rather than before it. The constant is no longer
gap-specific, so it is now named for what it asserts.
* perf(filer): re-park a gap wait woken by an ordinary append
Draining one stale token did not bound anything: the subscriber channel
carries an append per metadata write, so under continuous writes the next
one satisfies the wait immediately. During the write burst these stalls
come from, each parked subscriber ran the recovery loop at write rate
rather than the intended two-second cadence.
Only a flush can settle a gap, so check the flush watermark on wake-up and
go back to waiting if an append is all that arrived. The retry timer is
created once, so re-parking does not extend the interval.
* fix(filer): keep a disk-derived cursor inclusive
Marking every disk position exclusive assumed the entry at that timestamp
was the only one there. On the aggregated stream it is not: disk can
deliver one filer's persisted event at T while another filer's event at
the same T is still unflushed in the aggregate ring. The exclusive cursor
skipped it, and since the persisted reader also excludes ts <= its start,
nothing would ever bring it back.
Take the weaker guarantee instead. The reason the exclusive cursor was
introduced - an inclusive one landing on the eviction watermark parks
forever - is better answered in the resolvers: at the watermark memory
holds nothing (retained windows start strictly after it) and the persisted
reader cannot return that entry at any later time either, so refusing the
gap buys nothing and never ends. Skip it whatever the cursor's
inclusivity. That proof does not depend on a flush, so the local resolver
takes it as a second, independent disjunct alongside its flush watermark.
* perf(filer): wake a parked gap wait on flushes only
Re-parking on an append bounded the work but not the wake-ups: the
subscriber channel carries one per metadata write, so a parked subscriber
still took a scheduling round-trip per write. Worse, draining it kept the
channel empty, so every writer's non-blocking notify succeeded instead of
falling through - the burst paid for the wake-ups too.
Give the log buffer a flush-only subscriber list, notified from loopFlush,
and park on that. The append channel now fills once and stays full, which
is exactly the state the non-blocking send is designed for.
* fix(filer): stop the gap-stall gauge from leaking a series per connection
clientName is req.ClientName + "@" + peer address, so it carries the
client's ephemeral source port and changes on every reconnect. Labelling
the gauge with it minted a new series per connection, and clearing it only
ever Set(0), so nothing was ever released - a client in a reconnect loop
grows the filer's metric map and /metrics payload without bound. Key it on
the stable client-supplied name, as the neighbouring subscribe gauge
already does, and delete the series on teardown.
Two logging fixes ride along, both in the same reporter: the resume
warning was unpaced while the park warning throttles to one a minute, so a
burst that parks and resumes every couple of seconds warned on every
cycle; and clear() doubled as the teardown path, announcing "resumed
after 14m0s parked" for a client that actually gave up and disconnected
still behind.
* fix(filer): give a parked gap wait the exits the read loop has
A park never re-enters the read loop, so every exit that loop relies on
stopped working while a subscriber was parked. It kept scanning the filer
store every 2s for a client that a higher-epoch reconnect had already
superseded, until the TCP connection finally died - hours, on a half-open
one. It never reached the only code that honors UntilNs, so bounded
callers like `weed shell fs.verify` and `filer.meta.tail -until=` hung
instead of exiting. And a notification channel closed out from under it
turned the bounded wait into a spin, since a receive on a closed channel
returns instantly.
Check all three where the subscriber actually waits. Bound the wait too:
waiting is productive while the window is still queued for flush, but a
peer that never returns, or whose filer store this filer cannot read at
all, makes it permanent - and a subscriber that silently stops delivering
is no better than one that silently skips. Fail the stream after that
instead of hanging, loudly enough to say which gap and for how long.
* fix(log_buffer): keep the eviction gate out of the shared read path
The gate belonged to the filer's subscribe loops but was installed in
ReadFromBuffer, which the message queue shares and which has no gap
handling of its own. Three MQ paths broke on it: GetUnflushedMessages
asks for everything in memory past the flush watermark and got
ResumeFromDiskError instead, so SQL results silently dropped unflushed
rows; a RESET_TO_EARLIEST consumer's epoch cursor was sent to disk, and
the MQ disk reader resets an empty read back to epoch, so the whole
partition replayed on every pass; and a disk cursor landing on the
watermark carried offset -2, which the gate refused and the disk reader's
ts <= start filter also skips, so neither side could ever serve it. The
rewrite also dropped the old Offset <= 0 requirement, letting a stale
positive-offset cursor jump to memory over on-disk history, and refused
any negative-timestamp cursor even with nothing evicted.
Restore the read path exactly as it was and keep only lastEvictedTsNs,
which is the useful primitive. The filer loops now consult the watermark
themselves before reading memory, which is where the gap handling that
makes the refusal actionable already lives - and they check it every pass,
not just when the disk came up empty, since a disk read can leave the
cursor short of the watermark too.
* fix(filer): read the log file whose window spans past its own name
A log file is named for the start of the window it holds, but a window
runs up to a flush interval longer, so "12-30" can hold entries through
12:31:58. File selection compared the cursor's minute against that name
and skipped anything sorting earlier, so a subscriber resuming at 12:31:10
never saw the rest of that file: the read reported nothing on disk while
the entries sat in it.
That was survivable when a miss only meant "wait and retry", but the gap
resolvers now read a miss as proof the range is empty and move the cursor
past it, which turns those entries into silent loss. Start the file scan a
flush interval early; entries are still filtered against the exact cursor,
so this only opens one more file and never re-delivers.
* fix(filer): resume a gap with a cursor the memory read will serve
Reverting the eviction gate put ReadFromBuffer back to refusing every
positive-offset cursor below the in-memory window, but the resolvers still
handed back one - earliest-1 marked exclusive. So the resume bounced
straight to ResumeFromDiskError, the disk had nothing, and the resolver
saw its own cursor as no progress and parked: a subscriber stalled, and
after the new bound failed outright, with the whole gap sitting in the
ring the entire time.
The exclusive marking only existed to dodge a park the gate itself caused,
and the gate is gone. Resume at earliest with the sentinel offset, which
is the position master used and which case 2.1 reads inclusively. That
makes the resume unconditionally ahead of the cursor, so the progress
check it needed goes away with it.
* fix(filer): stop dropping a log file whose window outruns its name
Listing the earlier file was not enough: the iterator then decided whether
to read it by comparing the *following* file's name against the cursor,
which treats that name as an upper bound on this file's contents. It is
not one. A file is named for the start of the window it holds,
minute-truncated, and the window runs up to a flush interval longer, so
"12-30" can hold an event at 12:31:20 while "12-31" sits right after it -
and a cursor at 12:31:10 skipped straight past the event.
Bound the decision on the file itself: skip it only when its name plus the
minute truncation plus a flush interval still lands at or before the
cursor. That also covers the last file in the queue, which the old check
never skipped because it had no successor to compare against.
* fix(filer): stop a chunk-ref read from rewinding the subscriber
CollectLogFileRefs reports the minute-level name of the last file it
shipped, and the caller assigns that straight to the read position. Since
the scan now reaches back a flush interval to catch a spanning file, a
request at 12:31:10 that picks up the 12-30 ref moved the cursor to
12:30:00 - so the memory read that followed replayed events older than the
client's own SinceNs and re-sent what the chunk reader had already been
handed. The same rewind was reachable before, within a minute, whenever
the last file's name sorted behind the request.
Clamp the reported position to the one that was asked for. It still
under-advances by design, since the server never reads the entries it
ships refs for and cannot know where they end.
* fix(filer): resume below earliest so a one-entry window is not skipped
Moving the resume onto earliest itself was wrong for the smallest window
there is. A sealed window holding a single entry has startTime ==
stopTime, and the sealed-buffer lookup only enters a window whose stopTime
is strictly after the cursor, so a cursor sitting exactly on earliest
walks past it and its sole event is never delivered. Low-volume metadata
windows are routinely one entry, which is precisely when losing it is
hardest to notice.
Go back to one nanosecond below, which takes the startTime.After branch
and returns the whole window, and keep the sentinel offset the memory read
requires. The earlier test used an active multi-entry buffer, where both
cursors happen to work; the new one seals single-entry windows and
compares which entry comes back.
* fix(filer): count a persisted read as progress only when it moves
A chunk-ref read reports the minute-level name of the last file it
shipped, now clamped so it never rewinds, so it comes back non-zero even
when it names the position the subscriber already held. The loops read
non-zero as progress: they cleared the stall timer, then found the cursor
still short of the eviction watermark and parked again. Every retry
re-shipped the same refs and reset the timer, so the bound that is
supposed to end an unrecoverable stall was never reached - and a
chunk-capable client buffers those refs waiting for an event that never
comes, so it just accumulates duplicates.
Require the reported position to be strictly ahead of the cursor.
* fix(filer): count the evicted ranges the aggregated stream cannot prove
The eviction watermark belongs to the merged ring, but the disk it gets
checked against is the union of every peer's own log and each peer flushes
on its own schedule. A read that lifts the cursor from below the watermark
to above it may have done so entirely on a peer that is already ahead,
while a lagging peer still holds unflushed events inside the range just
crossed; when it flushes them they sit behind the cursor and are never
delivered. One aggregate maximum is not proof that every peer persisted
the range.
Nothing available locally separates that from the ordinary case where
every peer had in fact persisted it: the aggregator tracks peers by
address while log files carry a random per-filer id, so "has this peer
flushed through T" cannot be answered here at all. Deciding it needs the
source filer's own flush watermark carried on the subscribe stream, which
is a wire change this does not make. Count and log the crossing so the
window is at least measurable instead of invisible.
* fix(filer): send log file refs through the pipelined sender
sendLogFileRefs wrote on the raw gRPC stream while pipelinedSender's
goroutine concurrently calls Send for queued memory events - two senders
on one stream, which gRPC forbids. The window used to open once per
fall-behind; the gap retry loop now reopens it every pass. Routing refs
through the sender also restores ordering: refs used to overtake up to
1024 queued events, and the client treats any non-ref message as the
signal to process buffered refs, so overtaken refs were applied against
the wrong position.
The reason refs bypassed the sender was the batcher: their TsNs of 0
reads as far behind, and the client recognizes refs by the top-level
field alone - a refs envelope would drop its Events tail, and refs inside
Events would be applied as an empty event. Teach the batcher instead:
refs messages always go solo, and one drained mid-batch is sent solo
right after that batch.
* fix(filer): count parked subscribers instead of flagging them by name
The per-client gauge series could not work. Its label was rebuilt from the
peer address at first, which leaks a series per reconnect; keyed on the
client-supplied name instead, it collides - every mount registers as
"mount" - so one stream's teardown deleted a parked sibling's live series,
and the sibling never re-created it because its own park state said the
gauge was already set. Either way the alert this gauge exists to drive
goes dark.
A count needs no identity: Inc on park, Dec on resume or teardown, scope
as the only label. Client details stay in the logs.
Also start warning only once a stall has outlived the warn interval.
park() warned immediately on every first park, so catch-up churn that
parks and resumes every couple of seconds logged a warning pair per cycle
- exactly the flood the pacing was supposed to prevent, burying the
long-stall warnings that matter.
* fix(filer): one gap resolver, and re-arm an unservable adjacent cursor
The two resolvers were the same function - the aggregated one is the
local one with a flush watermark of zero, since its ring never flushes -
duplicated down to the comment justifying the resume target. A fix
applied to one and not the other is how the two streams drift; merge
them.
The merge carries the one behavioral fix both copies needed. Timestamp
collision bumps make adjacent entries exactly 1ns apart, so an entry
ending an evicted window leaves the cursor exactly one below earliest
with a positive batch offset. The resume target then equals the cursor
and both copies refused it as no progress - but that cursor cannot be
served (ReadFromBuffer refuses positive offsets below the window) while
the sentinel resume at the same timestamp is, and both deliver exactly
the entries after it. Refusing parked a subscriber whose data was
entirely in memory: until the next flush locally, and through a
15-minute stall failure on the aggregated path. Advance on equal target
when the held cursor is exclusive; a sentinel there is already served,
so it still refuses.
* fix(filer): a bounded subscription parked exactly on UntilNs is finished
The park's UntilNs check was strict while the bound is inclusive and
cursors are exclusive: a disk read whose last entry sits exactly on
UntilNs leaves the cursor there with everything up to the bound already
delivered. If the next range was an unprovable gap, the completed
subscription parked anyway and eventually failed - fs.verify hanging and
then erroring on a healthy cluster.
* fix(filer): re-derive the subscribe loops as one state machine
The loops had grown three generations of gap handling - a post-disk-read
branch, a post-memory-read branch, and an every-pass guard bolted on in
front of the memory read - each consulting state the others mutated. The
worst interaction wedged the aggregated stream permanently: diskExhausted
compared the disk result against the cursor that result had just updated,
and paired it with a ResumeFromDiskError latch that only a resolver
advance cleared, so one fall-behind sent every later pass into the gap
block and LoopProcessLogData never ran again. Mounts kept a
healthy-looking stream and applied nothing.
Both loops now run the same derived sequence. One disk pass; progress is
the pre-update cursor against the result, freshly each pass. One gap
decision before the memory read: a cursor the ring evicted past either
keeps draining the disk (it just advanced), resolves forward (the gap is
proven empty), or parks - and a cursor memory refused with nothing
evicted after it re-arms onto the retained window. The stale-latch branch
is gone: the error is only consulted on a pass whose own disk read came
up empty. All four parks go through one parkOnGap helper, so the exits
live in one place. The eviction watermark is read after the disk read and
the same value feeds both the guard and the unproven-crossing report,
which previously compared against a snapshot taken before a potentially
minutes-long backlog read and missed evictions landing during it. The
next-day jump now clears the stall reporter - it used to leave a stale
park epoch that could kill the next brief park instantly at the 15-minute
bound - and counts its own watermark crossing.
Chunk-ref reads get two rules the old loops lacked. Refs are sent once
per position: retries re-sent identical batches every two seconds into a
client that only drains them on a non-ref message, growing an unbounded
pending list. And when refs cannot advance the cursor - they report file
start minutes, which sit below the content the client was actually given,
pinning the cursor under the watermark forever during bursts - the pass
falls through to entry reads, which move the cursor by real timestamps
and double as the client's drain signal.
* fix(filer): give up on an unprovable gap instead of failing the stream
Failing after maxGapStall assumed the client could do something better,
but every consumer just reconnects at the same SinceNs and hits the same
wall, so an unprovable gap - a dead peer, or a peer whose filer store
this filer cannot read at all - turned into a permanent 15-minute
fail/reconnect loop delivering nothing. Master handled the same state by
skipping instantly and silently.
Take the middle: wait the full bound, then abandon the gap and resume at
the eviction watermark, where everything retained starts strictly after,
so the loss is exactly the range that could not be proven. The skip
shares the unproven-crossing counter and logs at error level - loss is
bounded, recorded, and the stream keeps working. A stall with nothing
evicted past the cursor loses nothing by waiting, so it restarts the
clock and keeps parking rather than skipping.
* test(filer): pin the file-skip bound through the production predicate
The spanning-file test asserted against its own copy of the arithmetic,
so a regression in the iterator - restoring the next-file-name comparison
or dropping the flush-interval term - would keep CI green while
re-introducing the silent loss the fix closed. Extract the bound into
logFileMayContainAfter, call it from the iterator, and point the test at
it; breaking the production expression now fails the test.
* test(log_buffer): pin the flush-subscriber contract
The registry the filer's gap parks wait on had no test at all. Cover the
observable contract: an append never wakes a flush subscriber, a flush
does with the watermark already stored, unregistering closes the channel
so an abandoned waiter unblocks, and double or unknown unregisters are
harmless.
The store-before-notify ordering in loopFlush is what makes the parks'
wake-up re-check sound, and it is not black-box testable - reordering
leaves a same-goroutine window of nanoseconds that hundreds of tight
round-trips never catch. Mark it load-bearing at the site instead; a
reorder now at least has to argue with the comment it deletes.
* fix(filer): a -1 SinceNs is a position, not the refs-gate sentinel
The once-per-position refs gate used -1 as "never sent", but a client may
legally subscribe with SinceNs=-1, whose cursor timestamp is exactly -1:
the very first pass then believed refs were already sent there and fell
back to streaming the whole persisted history entry by entry - the
bootstrap load chunks mode exists to avoid. Use MinInt64, which no cursor
can carry.
* refactor(filer): drop the aggregator's listener cond with no waiters left
Same shape as the FilerServer cond already removed: nothing increments
ListenersWaits and nothing ever calls Wait, so the three Broadcasts wake
nobody and the notify callback's guard is always false. Aggregated
subscribers wake through the buffer's subscriber channels now.
* fix(filer): make the gap metrics say what they count
The crossing counter's help text described only the aggregated peer case,
but give-ups increment it for local stalls too - a wedged local flush -
which sends an operator chasing peer replication when the problem is the
local store. Label it by scope and say both. The stalled gauge counted
every park, including waits with nothing evicted and nothing at risk,
while its help text promised evicted-but-unpersisted events; describe it
as what it is, a count of subscribers parked on a gap.
* test(filer): make the flush and stall tests assert what they claim
The flush-subscriber rounds were vacuous: the probe entry sat above the
round timestamps, so every round entry was collision-bumped and the
stored watermark exceeded the local value each assertion compared
against - the same bump mistake this test suite already made once. Put
the probe below the rounds and guard each round against bumping, so a
vacuous setup fails instead of passing.
The stall-outcome test wrote the reporter's park epoch directly,
bypassing the gauge Inc that gaveUp() later Decs - leaving the shared
process gauge at -1 for every test that runs after it. Park through the
real path, age the park by hand, release what the test holds, and assert
the gauge lands back where it started.
* fix(filer): finish the stream checks before marking it parked
parkOnGap stamped the reporter before waitOnGap ran its instant done
exits, so a bounded subscription completing inside a gap state was marked
parked for the microsecond before done fired - a phantom gauge blip and a
false "disconnected still behind" warning on every healthy completion.
Fold waitOnGap into parkOnGap so the done exits run first and the park
mark only ever covers a stream that actually waits.
While the park owns its timer, back the retry off as the stall ages -
2s probes growing toward one a minute - since every retry re-reads the
persisted log, and probing the store each 2s for 15 minutes per parked
subscriber during the very outage that parked them makes the bad time
worse. The subtest still named for the old fail-the-stream stall
behavior goes with the merge.
* fix(log_buffer): gate filer cursors against eviction under the read lock
The subscribe loops checked the eviction watermark and then read memory,
but a seal can land between the two: the read then served a sentinel
cursor from the earliest retained window, silently skipping the window
just evicted - the loss class this PR exists to make loud, surviving as
a race.
The only place the check is atomic with the serve decision is inside
ReadFromBuffer, under the lock seals take to evict. Rather than put the
policy back into the shared read path - which broke four message-queue
readers last time - add a new sentinel offset that opts into it:
EvictionGatedOffset reads inclusively exactly like -2, except below the
watermark it is refused to disk. The filer loops stamp it on every
cursor they hand the memory read; a refusal lands in the same gap
machinery the loop-side check feeds, so the race collapses into the
handled path. MQ cursors never carry it and keep master behavior
byte-for-byte.
* fix(filer): gate the aggregated gap on received, not bumped, timestamps
The aggregated ring rewrites an out-of-order arrival to its head plus a
nanosecond, so after any bump-heavy interval - a peer history replay
following a restart is enough - its eviction watermark lives above every
timestamp that exists on any peer's disk. Comparing a disk cursor against
it parked subscribers that had in fact drained every peer's log: a
15-minute delivery freeze ending in a give-up skip and a false loss
alarm, on a healthy cluster where master resumed instantly.
Track a second watermark in the received timestamp space - the highest
pre-bump timestamp among evicted entries - and gate the aggregated loop
on that. Disk cursors and received timestamps are the same space, so the
comparison means what it says: at or past it, every evicted entry's
original was at or below the cursor, and everything flushed of them was
already delivered. The bumped watermark keeps guarding the in-ring read
gate, whose cursors live in ring space. The local buffer is untouched:
it flushes its own bumped timestamps, so there the two spaces are one.
* fix(filer): ship each log chunk once and stop echoing ref'd files inline
Chunk mode duplicated data through two doors. Consecutive ref
collections overlap by design - the scan backs off a flush interval to
catch a spanning file, and a filer appends chunks to its newest file -
so the same file was shipped again on every pass that re-listed it: the
client re-downloaded its chunks, and a duplicated file mid-batch rewinds
timestamps inside the client's per-filer merge, which reads each stream
as sorted - transiently resurrecting deleted entries during catch-up.
Track per subscription how many chunks of each file were shipped and
send only the unsent suffix; state prunes with the scan window, so it
holds a few files per filer.
The second door was the entry fallback: when refs cannot advance the
minute-named cursor, the pass streamed the ref'd file's tail inline, and
the client applies inline events unfiltered - the same tail it already
applied from chunks. Entry passes for chunk clients now advance the
cursor without delivering; everything they skip is covered by the refs
already sent or the deltas the next collection ships.
* refactor(filer): one gap decision shared by both subscribe loops
The post-disk gap tree - guard, drain, resolve, two parks, the re-arm -
existed twice, differing only in buffer, watermark space, flush getter,
park channels, and reason strings. Four rounds of review fixes have shown
the copies drift the moment one is edited alone. gapPass now carries the
five differences and the tree lives once; the loops shrink to a
three-way switch between reading memory, restarting the pass, and ending
the stream.
* docs(filer): trim the gap-machinery comments to the why
Several blocks had grown to ten-plus lines restating what the tests
already pin or retelling one rationale at multiple sites. Keep the
non-obvious why - the load-bearing flush ordering, the two timestamp
spaces, the refusal-at-equality argument - in a few lines each.
* fix(log_buffer): credit an entry's received timestamp to its own window
The received-ts capture ran before the rollover check, so an append that
sealed the previous window stamped its timestamp onto that window and
then lost it in the reset of the new one. The eviction watermark this
feeds broke both ways: the sealed window's value was inflated by an entry
it does not contain - parking aggregated subscribers on gaps that were
drained - and the entry's real window was deflated, proving gaps empty
that still held its event on some peer's unflushed path. Credit the
timestamp only after the entry lands, when its window is known.
* fix(filer): rebase a shipped chunk suffix to logical offset zero
A grown file's delta kept the chunks' original file offsets, but the
client's chunk reader starts at logical zero and a list opening higher
reads as instant EOF - a successfully empty replay, and since chunk
clients no longer receive disk entries inline, the appended events were
silently dropped. Clone the suffix chunks with offsets rebased to zero;
the cut is record-aligned because each append is one uploaded chunk of
whole entries, so the suffix decodes as a file of its own.
* fix(filer): finish every chunk refs batch with a transition the client acts on
Both chunk consumers buffer refs until a non-ref message arrives, so a
source with historical logs and a quiet ring - a mount reconnecting
after a filer restart is the common case - shipped its backlog and then
went silent: the client sat on the refs until the next metadata mutation
anywhere in the cluster. The disk step now ends every batch with the
empty-notification marker the client already treats as a resume-cursor
advance.
The same step closes the inline replay: the cursor used to stay at the
last file's minute name, so the memory read re-delivered the retained
tail of a file the client had just read via chunks - T1..Tn applied
twice. The advance-only entry read now runs on every chunk pass, moving
the cursor to the true disk content end before memory is consulted.
Ordering inside the pass is load-bearing: the entry read can outrun the
shipped refs by a chunk appended between collection and read, and the
transition timestamp becomes the client's refs filter - stamping it past
unshipped content would silently drop that chunk's events on the next
delta. The pass therefore re-ships the delta after the entry read, so
the transition never exceeds shipped content. Bump-displaced aggregated
entries can still arrive inline above the cursor with originals below
it; that duplication is bounded and stays within the documented
at-least-once residual.
* fix(filer): prune ref state at the minute the scan actually stops at
The collector compares file names at minute granularity while the prune
used the exact-nanosecond scan bound, so for a cursor at 12:31:20 the
12-30 file was still collected but its sent state was already deleted -
the next pass reshipped the whole file, re-creating the duplicate-refs
class the state exists to prevent. Truncate the bound to the minute the
file names live in.
* fix(filer): derive the chunk cursor from the shipped refs themselves
The advance-only entry read left the three positions that must agree in
each other's blind spots. Its snapshot could trail the second delta's, so
a chunk appended between them shipped events newer than the cursor and
the memory pass sent them again. And it made the filer decode the tail
range on every pass, serialized ahead of the client's own reads by the
transition marker - re-introducing a slice of the replay work chunk mode
exists to offload.
Compute the cursor from the shipped set instead: the final entry
timestamp of each filer's last shipped chunk, decoded once through the
shared chunk cache. Refs coverage, transition marker, and memory start
are then the same number by construction - nothing is decoded twice,
nothing is dropped, and the per-pass server cost falls to one cached
chunk decode per filer. The second delta and the once-per-position refs
gate existed to patch the entry read's snapshot races, so both go with
it; the range read survives only as a fallback for legacy chunks that do
not decode standalone.
* fix(filer): keep the chunk-cursor probe inside the shipped snapshot
Three holes in the tail probe, all variations of stepping outside what
was shipped. A permanently missing chunk failed the stream before the
transition marker, so the client discarded its pending refs and
reconnected to the same failure forever - blocking all later metadata
behind one dead volume, where every other replay path (including the
client's own reader) skips such chunks; the probe now walks back to the
last readable chunk, and a filer with nothing readable simply contributes
no cursor. The legacy fallback re-listed the logs after the refs were
collected, so a concurrent append could push the range end over an
unshipped chunk and the marker past events the client never received; it
now streams the shipped chunk list itself, so no snapshot other than the
shipped one is ever consulted. And a file selected before UntilNs can
hold entries past it, which the client filters while still adopting the
marker as its checkpoint - a later bounded request then skipped them;
the marker is clamped to the bound.
* fix(filer): make the cursor probe an exact mirror of the client's reader
The probe answered from the server's view of the chunks; the marker's
correctness depends on the client's. Its backward walk found the last
readable chunk, but the client reads forward and stops at the first
unreadable one, never resuming within a file - for readable, missing,
readable the marker claimed the suffix the client never applied, losing
those entries permanently. Keeping only each filer's final file ref
discarded the progress of earlier readable files when that file was
wholly missing, rewinding the marker to the start cursor. And a torn
trailing size prefix - what a crashed writer leaves - failed the probe
where the client reads a clean end, blocking the marker forever on data
the client accepts.
The probe is now shaped like the reader it answers for: per file the
readable prefix, per filer the newest file with content, and no
condition escapes as an error - understating the marker only re-ships,
overstating loses events, and a probe failure must never block the
transition the client is waiting on. Each rule is pinned by a test that
fails against the previous shape.
* fix(filer): judge chunk readability at the volumes, not the decode cache
Two ways the probe's answer could drift from what the client experiences.
A chunk this server decoded earlier stays warm in the shared cache after
its volume dies, so the probe sailed past a chunk the direct-reading
client stops at - marker beyond the unread suffix, entries lost. Every
chunk now passes a volume lookup before the cache is consulted; the
lookup rides the master client's in-memory map, so the probe stays cheap.
And a probe stop was treated as harmless understatement, but the delta
had already marked the whole ref sent: a transient server-side failure
left the cursor stranded behind shipped content for the life of the
connection, parking aggregated streams below the watermark for data the
client already holds. The pass now rolls back the sent state of every
ref above the file that answered the probe, so unreached refs re-ship
and re-probe until the cursor gets there. Re-shipped entries at or below
the client's checkpoint are filtered client-side, and batches are
marker-separated, so a re-shipped file cannot rewind a merge mid-batch.
* test(filer): end-to-end subscribe-loop harness and wire-contract tests
Every escaped bug across this change's review rounds lived in an
interaction the unit tests could not see: the loop state machine, the
disk/memory handoff, or the server/client contract. The harness runs the
real SubscribeLocalMetadata loop against a real leveldb-backed filer,
faking only the volume layer behind the existing test hooks, and asserts
the delivered stream itself.
Eight scenarios, each pinning a class this change was reviewed for: the
headline evicted-unflushed gap parks and then delivers in full; a ring
that evicted nothing serves memory promptly; a backlog-to-live handoff
with 1ms-adjacent timestamps across every boundary delivers exactly
once; a flush-proven gap over vacuumed log files skips to the retained
ring including a single-entry window; a bounded subscription terminates
at its bound; a permanently wedged flush ends in the give-up skip with
the stream still alive; and chunk mode is checked against the real
client code - pb.ReadLogFileRefs applied to the shipped refs must cover
everything the transition marker claims, with and without a dead volume
in the middle.
Validated by re-introducing three fixed bugs: the missing eviction guard
delivers during the unproven gap, a 2ms cursor error at the handoff
drops exactly one event, and resuming at rather than below the earliest
retained window loses a single-entry window's sole event - each caught
by the scenario built for it. The gap timing knobs become vars so parks
run at test speed, a small filer hook swaps the volume-touching read
functions, and a sender test pins the refs wire rules the client
depends on: never batched, never an envelope, everything in order.
* fix(filer): re-ship a partially read answering file, pin the probe's limits
The sent-state rollback stopped at files newer than the one that answered
the probe. When the answering file itself was only prefix-readable - a
dead or transient chunk mid-file - its unread suffix stayed marked sent,
and the next append advanced the cursor past it for good. The probe now
reports whether the answering file was read through to its end, and a
prefix-limited answer re-ships that file too; a torn tail counts as
complete, since the client's read ends there as well. The rollback rules
live in one predicate with a table test - files below a complete answer
stay sent, because the client has moved past them and re-shipping cannot
rewind its filter.
Two test honesty fixes ride along. The loop harness derived its timestamp
base from time.Now() per call, so expectations recomputed across a second
boundary drifted by exactly one second; the base is now fixed per
harness. And the probe's liveness boundary is pinned as a test instead of
a comment: a volume lookup cannot see a dead needle or a stale location
inside a resolvable volume, so a warm cache can answer past a chunk the
client fails on - accepted because metadata log chunks die
volume-at-a-time and the alternative is a real read per probe, which is
what the probe exists to avoid. The test states the boundary so changing
it is a decision, not an accident.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3 remote: honor s3.support_tagging in UpdateFileMetadata
The write path already skips tagging when the remote is configured
without tagging support, but the metadata-update path sent
PutObjectTagging or DeleteObjectTagging unconditionally. On remotes
that reject tagging requests, any metadata update failed -- including
updates with no tags at all, which land on the DeleteObjectTagging
branch.
* s3 remote: drop the never-read supportTagging field
Every maker set it, nothing read it: the tagging decision is made from
conf.S3SupportTagging. Keeping a field that looks like the switch but
is not invites exactly the inconsistency the previous commit fixed.
* remote storage: actually delete objects when a directory is removed
On object-store backends RemoveDirectory returned nil without doing
anything, so a directory delete synced to the remote as a successful
no-op and the objects under that prefix stayed there forever. Nothing
surfaced the divergence: the sync logged rmdir, advanced its offset,
and the local namespace looked clean.
Deleting a bucket-level directory on a filer store that can drop a
whole bucket emits no per-child delete events at all, so the single
rmdir event was the only chance to clean up the remote.
Each backend now lists the prefix and deletes what it finds: S3 in
DeleteObjects batches of one listing page, GCS and Azure per object.
The prefix always ends with a slash so a sibling like dir2 survives
deleting dir, and errors propagate so a failed delete is retried
instead of silently skipped. A directory that maps to the bucket root
is left alone: wiping every object in the bucket from one namespace
event is too destructive, and bucket removal already has its own path.
* gcs remote: wrap the per-object delete error
The listing error in the same function already wraps, so the delete
error should stay inspectable with errors.Is as well.
* s3 remote: name the empty-listing test for what it checks
The prefix in that test is a normal directory; what is empty is the
listing. The bucket-root guard has its own test.
* s3 remote: report the scope of a failed delete batch
A DeleteObjects response can carry per-key errors for up to a
thousand keys. Surfacing only the first hid how much of the batch
failed, and surfacing all of them would build an unbounded error
string, so report the count with the first failure as the sample.
Shutdown drops the sealed-chunk map references, but an in-flight
uploader goroutine holds the final reference and releases its budget
slot only after reacquiring chunksLock. Asserting Used()==0 immediately
after Shutdown races those releases on slow runners. Poll with a bounded
deadline instead.
A zero-data needle lands in .dat as a size-0 record, byte-identical to
a delete marker, so scans that walk .dat count it as deleted. Once in
1024 writes newRandomNeedle produced one, and the idx-head repair then
skipped a row TestRepairIdxHeadTombstones_ReadOnlyVolume expected back.
* s3: test the GetCallerIdentity handler
The handler had no test, only XML marshalling, so nothing pinned that a
caller presenting session credentials is reported as the assumed role
rather than the user who minted the session.
* s3: drive AssumeRoleWithWebIdentity over HTTP with a real OIDC token
Coverage reached the OIDC path either at the IAMManager service layer or
through the Authorization: Bearer shortcut. Nothing exercised the public
STS entry point an AWS SDK actually calls, which is where parameter
parsing, the IAMManager dispatch and the XML response shape live.
* s3: test that every STS route emits an audit entry
STS responses go out through WriteXMLResponse, which never calls PostLog,
so track() is the only thing that logs them. A route registered outside it
would mint credentials with no audit trail and nothing would notice. STS
has three routing layers, so a new action is easy to attach to the wrong
one.
* s3: make the STS tests assert what they claim to cover
The audit routing test ran against an uninitialized STS service, so every
case answered 503 and a non-404 status was the only evidence the request
had reached STS at all - the POST-body case could have been served by the
dispatcher's IAM branch and still passed. Back it with a real STS service
and assert the STS response namespace, which IAM and S3 responses do not
carry.
The session policy case checked that the policy travelled in the token
rather than that it restricted anything; assert the narrowed bucket is
allowed and another bucket is refused.
Give the forged-token case the same claim set a valid token gets, so it
cannot pass for want of a claim, and cover both rejection paths: a key we
do not publish, and a key id absent from the JWKS.
The crowded map keeps volumes that later became unwritable, so their
state survives transient writability flips without flapping. But volumes
packed to capacity (fs.mergeVolumes) or turned read-only stay above the
crowded threshold and get re-marked on every heartbeat, so the raw map
size can permanently exceed the writable count. ShouldGrowVolumes then
returns true forever, every assign-path grow request passes the gate,
and the periodic grow loop fires too, creating volumes without bound --
worse with -volumePreallocate.
Count crowded as the intersection with writables instead: growth checks
and the layout gauges only see crowded volumes that can still take
writes.
* s3: log the requester's principal ARN in the audit entry
An STS session authenticates as an opaque session subject, so requester
alone gave an operator no way back to the assumed role or the session
name. Record the principal ARN next to the identity name and emit it as
requester_arn.
* s3: record the caller identity in the STS handlers
AssumeRole, GetFederationToken and GetCallerIdentity verify the caller
themselves and are not wrapped by the auth middleware that records the
identity, so every audit entry for minting a session had an empty
requester.
* s3: resolve the audit principal ARN the way policy evaluation does
A JWT-authenticated identity carries no PrincipalArn — the auth layer
hands the principal over in a request header — so reading the field
directly left requester_arn empty for OIDC callers. buildPrincipalARN is
the resolver the policy path already uses: header first, then the
identity's own ARN, then a synthesized user ARN for legacy identities
that have none.
* s3: keep an admin's role session scoped to the role
AssumeRole copied the caller's admin standing into the minted session as
the is_admin claim, which short-circuits base policy evaluation. An admin
assuming a scoped-down role therefore kept full access and the role's
attached policies, explicit denies included, were never evaluated.
Only a session the caller assumed for itself carries the claim now — a
legacy static admin has no IAM policies for such a session to inherit.
* s3: name the caller when it assumes a session for itself
An identity that carries no principal ARN left the self-assumed session
with an empty role name in its assumed-role ARN. callerPrincipalArn
synthesizes the canonical user ARN for that case.
* test: cover listings spanning a run of retracted keys
A listing drops entries whose current version is a delete marker. When a run
of consecutive entries drops out, the page being filled can come back empty,
and an empty page is easily mistaken for the end of the listing — everything
after the run then never appears and the caller is told those objects do not
exist.
Backup repositories produce exactly this shape: a batch of keys under one
prefix is retracted while writing continues under the next.
Covers a retracted run before live keys and between live keys, walked with
page sizes smaller than the run so at least one page is filled entirely from
entries that get dropped, plus the version view of the same namespace where
every version and every delete marker must still be reported.
* test: sweep every page size in both walks and paginate the version listing
A listing on a versioned bucket is served from metadata cached on the
.versions directory entry so the whole listing is a single scan. The cache
carried size, mtime, ETag, owner and the delete-marker flag but not the
storage class, so newListEntry found none and fell back to STANDARD.
The result was that HEAD and the listings disagreed about the same object:
HEAD reported the class the object was stored with, while ListObjectsV2 and
ListObjectVersions reported STANDARD for every object. Clients that filter or
tier on storage class act on the listing.
Caches the class alongside the other listing fields, clears it with them, and
copies it in the routed RECOMPUTE_LATEST path so both finalize paths agree.
* test: compare ListObjects and ListObjectVersions over the same namespace
The two listings walk the same tree through separate code paths, so a client
navigating by versioned listings can see a different namespace than one
navigating by plain listings, and concludes keys are missing that are plainly
there. Testing each path on its own never catches that; only comparing them
does, and nothing compared them.
Asserts both report identical current keys and identical common prefixes
across a backup-shaped tree: nested prefixes, a prefix naming an object
exactly, a key that is simultaneously an object and the parent of other keys,
a partial key fragment, and a prefix matching nothing.
The version view is reduced to what a plain listing reports — latest versions
that are not delete markers — so the comparison is like for like.
* test: guard against truncated pages and cover the delete-marker path
* test: pin verb parity on lock-arbitration keys through acquire and release
Backup clients arbitrate repository ownership by writing and retracting small
keys under a fixed prefix and re-probing them, each probe using a different
verb. They trust those verbs to agree; a key reported present by one and
absent by another makes the client either spin or declare the repository
corrupt, and neither shows up as an error on the storage side because each
individual answer is locally correct.
The keys are written and immediately deleted by version id, which is the cycle
that empties a version container, so parity is asserted on both sides of the
delete and across repeated re-acquire cycles where residue accumulates.
Reports which verbs disagreed rather than just failing.
* test: run the reacquire cycle on every lock key, and drain probe bodies
* test: pin that an unusable version id is refused, never resolved
A version id containing a path separator, or "." / "..", can never name a
stored version. Resolving one to the null or latest version instead would let
a caller destroy a live version by asking for one that does not exist, on a
bucket configured for immutability.
The guard exists today and holds on every verb; it had no test. Pins two
properties: such a request is refused with a client error rather than a 5xx
(a 5xx invites endless retries of something that can never succeed), and the
version that does exist survives every refused request.
* test: require exactly 400 for an unusable version id
* test: cover delete idempotency on versioned object-locked buckets
Backup clients probe and retract lock keys continuously, so they routinely
delete keys and versions that are already gone, and they batch those deletes
alongside keys that do exist. S3 makes all of that succeed; returning an error
turns ordinary lock arbitration into a job failure.
The behaviour is correct today but had no coverage, and it runs through the
object-lock retention check, which is the most likely place for a missing
object to start being reported as an error.
Covers: deleting a key that never existed, deleting a well-formed version id
that names nothing (twice, and without disturbing the version that does
exist), and a batch whose middle key is missing — every requested key must
come back under its own name rather than silently taking another row's slot.
* test: verify the deletes actually took effect, not just that they returned
* s3: report a peer that went away as ClientDisconnected, not IncompleteBody
A streaming PUT whose body ends early is always reported as IncompleteBody
(400). That collapses two cases with opposite causes: the peer vanished
mid-upload, and the peer sent fewer bytes than it promised while still
connected. The first points at the network path, the second at the client,
and once merged they cannot be told apart from the logs.
Split out ClientDisconnected (499) and select it when the request context
shows the peer is gone. The upload itself keeps running on a background
context so chunks still finish, which means cancellation races the read
error; a missed signal degrades to IncompleteBody exactly as before.
* s3: note what request-context cancellation is taken to mean
* helm: give the bucket hook the credentials weed shell needs
The post-install bucket hook pipes s3.configure into `weed shell`. Since
#9442 the filer only serves its IAM gRPC service to callers presenting an
admin-signed JWT, and since #9536 `weed shell` mints that token itself -
but only if it can find the filer signing key. The hook job sees neither
a security.toml nor the WEED_* environment overrides: its env list is
hardcoded, and it is the only workload in the chart without a
security.toml mount.
So with jwtSigning.filerWrite=true the hook logs
error: failed to get user anonymous: rpc error: code = Unauthenticated
desc = missing authorization metadata
and `weed shell` exits 0 regardless, which leaves the Job green while
anonymousRead is never applied.
Render the merged extraEnvironmentVars into the job's env - keeping
non-string values as valueFrom, so keys held in a Secret stay a
reference - and mount security.toml the way every other workload does.
The hardcoded WEED_CLUSTER_* entries go away because those values are
part of the global extraEnvironmentVars defaults and would otherwise
render twice.
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: report a missing bucket hook Job instead of crashing
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: assert where the bucket hook mounts security.toml
A mount under the wrong path leaves weed shell without the key just as
surely as no mount at all, so check the target, not only the name.
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: keep the bucket hook's cluster endpoints chart-computed
Dropping the hardcoded WEED_CLUSTER_* left the readiness waits
dereferencing $WEED_CLUSTER_SW_*, which exist only while the values keep
the default alias. Rename the cluster, or clear extraEnvironmentVars, and
the hook polls "http://" for five minutes and then fails the release.
Derive the env names from the alias and render the addresses from the
chart, as all-in-one already does.
* helm ci: assert the bucket hook follows a renamed cluster alias
Renames the cluster and drops the default addresses, then checks that
every env name the readiness waits dereference is set, and that the
address is the chart's rather than the one left in the values.
* helm ci: check the hook's endpoints and security.toml source
The bucket hook tests took two shortcuts a wrong render slips through.
The alias test only rejected the stale master address the values kept, so
any other wrong address passed and the filer address was never compared at
all. And the security.toml test matched the volume by name, which a
same-named emptyDir or a foreign ConfigMap satisfies while `weed shell`
still has no signing key.
Compare both cluster addresses against the Services the chart renders, in
the default and the renamed-alias case, and assert the volume is backed by
the chart's security-config ConfigMap. Checked by pointing the volume at
another ConfigMap and the filer env at a wrong but non-empty address:
both now fail, both passed before.
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* wdclient: age vid map entries by generation instead of chaining snapshots
The vid map kept its history as a linked list of past snapshots, trimmed
in place by storing nil into a node's cache pointer. That cost up to six
full copies of the volume-location map, a recursive walk taking a
different lock per level, and deletes that had to cascade through every
generation. It also had to special-case explicitly-empty entries, or
fallback would resurrect locations a newer snapshot had cleared.
Keep one map instead, and stamp each entry with the generation it was
learned in. resetVidMap bumps the generation and drops entries that were
not relearned within the retained window, which is the same retention
the chain provided: an entry survives DefaultVidMapCacheSize resets.
The first write of a generation replaces an entry rather than merging
into it, so a volume that moved answers with where it is now — the
property a fresh map per reset used to give for free. Entries are
copy-on-write, so locations handed to a caller are no longer shifted
underneath it by a concurrent delete.
The map is never swapped now, so the client-side lock and its stable /
current accessors go away with it.
* wdclient: make vid map entries immutable and drop them once emptied
Review follow-up. Updating an entry in place left the copy-on-write
guarantee resting on callers never holding the entry pointer; install a
new entry instead, so the rule is simply that a stored entry never
changes.
Deleting a volume's last location now drops the entry rather than
keeping an empty one, which a client that never resets would otherwise
hold for every volume it ever saw deleted. Lookups already treat an
empty entry as a miss, so nothing observable changes.
* wdclient: let the newest generation decide between regular and EC locations
GetLocations checked the regular map first whatever its generation, so a
volume that was EC encoded kept answering with the regular copies the
previous master knew until they expired — for as long as the retained
window, since nothing relearns a copy that no longer exists.
The snapshot chain did not have this problem: the newest map was
consulted first and only a volume it knew nothing about fell through to
older ones. Restore that by comparing generations, with regular copies
winning a tie, since a tie means one generation reported both.
Sealing a logic chunk index that already held a sealed chunk dropped the
old chunk's only reference and freed its page chunk. That chunk's upload
may not have started reading it yet — Execute() returns as soon as the
job is handed to a goroutine — so mem.Free could hand a live 2 MiB mem
chunk back to the slot pool, the next NewMemChunk would overwrite it,
and the in-flight upload shipped whatever bytes were there. Under fio
randwrite the volume server rejected those needles with "Content-MD5 did
not match md5 of file data" and the FUSE write failed with EIO.
Give the sealed chunk a second reference for its upload, dropped only by
the upload itself, and let the upload unindex itself only while it still
owns the index — the unconditional delete could evict a newer sealed
chunk and hide its dirty pages from readers.
resetVidMap trims the cache chain by storing nil into a node's cache
pointer once it ages past vidMapCacheSize. A lookup that missed in its
own map and only then loaded that pointer could find the link already
severed, reporting "not found" for a volume that stayed resolvable the
whole time.
Load the link first, while it is still guaranteed live. A published
vidMap's cache pointer only ever goes from its ancestor to nil, so
reading it earlier can never yield staler history.
* filer: let a nested path rule turn worm off
mergePathConf ORs the booleans, so worm set on a bucket could never be
lifted on a directory under it, while every string field is overridden by
the more specific rule. Make worm tri-state instead: unset inherits, set
wins. readOnly, fsync and disableChunkDeletion keep the OR, so a nested
rule still cannot escape a lock the bucket set.
Configurations written before this carry an explicit "worm": false on
every rule, because they are marshalled with EmitUnpopulated. Reading
those back as an override would quietly drop worm from nested paths, so
filer.conf is now stamped with a version and the flag is dropped to unset
when the version predates it.
* filer: copy the worm value out of the matched rule
mergePathConf aliased the pointer into the merged result, so a caller that
wrote through it would reach into the stored rule.
* test: pin that a .vif replication outranks the superblock
Store.ConfigureVolume rewrites the .vif and never the replica-placement byte in
the .dat, so that byte keeps whatever the volume was created with for good.
readSuperBlock reads it and then overrides it from the .vif, which is what makes
a replication change take effect and survive a remount.
Invert that and every replication change silently reverts on the next mount,
while the .vif on disk still records what the operator asked for -- a durability
setting quietly going back to its old value, with nothing to indicate it.
Worth pinning rather than reading off the code, because the field beside it
resolves the other way: version takes the superblock over the .vif. Two fields,
one function, opposite precedence, each a line to invert wrongly.
Covers the empty case too, since a .vif that declares no replication has to
leave the superblock standing or a volume whose replication was never
configured would be forced to whatever the zero value parses as.
* test: drop the unreachable nil check on MaybeLoadVolumeInfo
It initialises the returned pointer before the existence check and every
return is naked, so it never yields nil. Guarding against it implied a
contract the callee does not have.
The master could die with a nil pointer dereference inside
raft.(*server).TakeSnapshot. The leader and follower loops ask for a
snapshot on every iteration and callers can ask at any time, so several
goroutines built one at once, all writing to the one pendingSnapshot
field. Whichever finished first saved it and set the field to nil, and
the rest either dereferenced nil while attaching peers and state, or
stored nil as the current snapshot, leaving the master with no snapshot
to send to a lagging peer.
v1.2.0 holds a mutex for the length of a snapshot and keeps the one being
built in a local. It also stops a snapshot recovery that cannot finish
from being reported as done, writes snapshots to one side and renames
them into place, loads the snapshot covering the most of the log rather
than the last one in filename order, and takes the lock that guards the
peer map on both sides, where a snapshot cloning the peers alongside a
membership change was a concurrent map iteration and write.
* s3: allow copying an object onto itself in a versioned bucket
The copy writes a new version instead of overwriting in place, which is how an
earlier version is restored. Buckets with versioning off or suspended keep
rejecting a self-copy that changes nothing.
* s3: cover the suspended-versioning self-copy rejection
Suspended versioning overwrites the null version in place, so a self-copy that
changes nothing stays rejected. Pin that alongside the never-versioned case.
* s3: keep the list marker exclusive for versioned objects
A versioned object lives in a "<key>.versions" directory, so the entry name
never matched the marker and start-after/marker returned the marker key itself.
* s3: match the list marker against the raw entry name too
A backend that echoes the marker it was given returns the ".versions" directory
name, which no longer matched once the comparison used the object name alone.
Cover both, and unit test each half.
TestProxyReadDropsCallerJwtQueryParam mints a read token up front and requires
the token the volume server would evaluate to equal it byte for byte. The expiry
claim has one-second resolution -- GenJwtForVolumeServer sets it from
jwt.NewNumericDate(time.Now().Add(...)) -- so two mints on either side of a tick
produce different strings for the same authority and the same file, and the
assertion fails for a reason the test is not about.
It surfaces on the 32-bit job, where the runner is slow enough that the HEAD
subtest (the second one, after a full proxy round trip) lands in a later second
than the mint at the top of the test. Confirmed directly: minting the same file
id with the same key either side of a boundary yields different tokens.
Assert what the test is actually about instead -- that the credential decodes
against the read key and authorizes this file id -- which holds whatever second
it is minted in, and is a closer statement of the property than string equality.
* helm: optional NetworkPolicy per component
In a namespace with a default-deny policy the chart cannot be installed:
the components never reach each other, and the post-install bucket hook
waits on the master and filer until it gives up.
networkPolicy.enabled renders one policy per component, selecting its
pods by the standard app.kubernetes.io labels and admitting the other
pods of the release on the ports that component listens on. The port
lists come from the same values as the containerPorts, and CI asserts
the two agree. Restricting egress is a second opt-in with extraEgress
for the filer store and notification sinks, which the chart cannot
know about.
Closes#10421
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: say "changed" instead of "retuned" in the policy comments
codespell reads "retuned" as a misspelling of "returned" and fails the
spelling job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: refuse empty port and DNS peer lists instead of widening
In a NetworkPolicy an empty ports list means every port and a missing peer
selector means every pod, so `kubeApiServer.ports: []` silently opened the
API server CIDRs on all ports, and nulling a DNS selector rendered
`podSelector: null`, which is every pod in kube-system. Both now fail the
render, and the DNS rule emits only the selectors that are set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: gate the bucket hook Job and its policy on one helper
Both were deriving the same condition from the same values, kept in step by
a comment. seaweedfs.bucketHookEnabled makes it one definition, so adding an
S3 mode cannot leave the Job running without its policy - which under
default-deny means the hook hangs. CI pins the pairing across the eleven
modes that decide it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: say that an egress default-deny needs both toggles
networkPolicy.enabled on its own only covers a default-deny that restricts
ingress. Where Egress is in its policyTypes as well, which is the usual
baseline, the components still cannot resolve DNS and egress.enabled is
required too. Both values and the README said the first half of that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: quote the label values the policies emit
The same int-coercion the hook templates were fixed for, in the file this PR
adds: unquoted, a release named 123 renders app.kubernetes.io/instance as a
YAML integer in the metadata, the podSelector and both peer selectors, and
the API server rejects the object - a policy that silently never applies.
CI now renders the chart as release "123" and fails if any policy label
comes out as a non-string.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: gate the resize hook policy on the same lookup as its Job
The policy rendered whenever the hook was enabled, which is the default, so
every release carrying networkPolicy got one - as a pre-install hook that
Helm never collects, left in the namespace after uninstall. It also made
egress.kubeApiServer.cidrs mandatory for every release, since the policy
claims the API server, for a Job that only runs on an upgrade that grows a
PVC.
Move the command computation the Job is gated on into a helper and read it
from both.
* helm: drop the API server rule from the admin policy
No seaweedfs binary talks to the Kubernetes API - there is no client-go in
go.mod - so the rule granted admin an egress path it never uses, and forced
anyone running admin with egress on to name an API server address for it.
The pod-RW ClusterRole the comment was reasoning from is a leftover from a
migration and is not read by any component.
* helm: reject a networkPolicy.components key that names no component
The component names are not guessable - objectstorage-provisioner,
seaweedfs-all-in-one, volume-<name> - and a typo silently dropped the rules
it was carrying. Check against every component the chart can produce, not
the enabled ones, so a values file shared across releases can still hold
overrides for a component this one leaves off.
* helm: name the all-in-one policy after the workload
componentName prefixes the release fullname onto the suffix it is given, and
the component label already starts with seaweedfs-, so the policy came out as
<release>-seaweedfs-seaweedfs-all-in-one.
* helm ci: assert the denied probe failed rather than that it did not succeed
kubectl run's "pod/x created" was captured alongside the pod log, so an
empty log would still not match exit=0 and the denial would pass without
anything having been tested.
* helm: document what turning the network policies on costs
Three things the values did not say: a Prometheus outside the release stops
scraping and nothing reports it, the resize hook's policy is a hook resource
that uninstall leaves behind, and the DNS selectors are wrong on OpenShift.
* helm: spell out the managed distributions codespell reads as a typo
codespell has AKS in its dictionary as a misspelling of ASK, so the DNS
selector note failed the spelling job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: read the probe verdict from the marker line, not the whole log
The default-deny step decides both probes from the pod log, but kubectl logs
returns stderr as well, and busybox wget reports "download timed out" there
even under -q. The denied probe therefore produced two lines beginning with
wget:, which matches neither exit=0 nor exit=*, so a correct denial landed in
the arm meant for a probe that produced no result at all and failed the job.
The earlier form hid this behind a catch-all that treated anything without
exit=0 as a denial; tightening that assertion made the stray line fatal
without narrowing the input it reads.
Pick the marker line out instead. The trailing || true is required: the step
runs under bash -e, so a grep that matches nothing would abort it rather than
reach the arm that reports an empty result, which is the case that assertion
exists to catch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* webdav: answer PROPFIND child stats from the listing
golang.org/x/net/webdav discards the FileInfo that Readdir returned and stats
every child again, five times over, so a PROPFIND on a directory costs five
sequential filer lookups per entry: 15006 lookups and 1.4s for 3000
subdirectories, 24s for 60000. Windows Explorer times out well before that.
Hold the entries a listing already fetched for the lifetime of the request and
serve those stats from them - 6 lookups and 0.03s for the same 3000 entries.
WebDavFile.Stat has to stop dropping its request context for the held entries
to be reachable.
* webdav: keep the request context on the lookups a listing drives
stat, Readdir and Seek reached the filer on context.Background(), so a client
that walked away left the listing streaming and the lookups running. Seek also
missed the entries the listing had already fetched.
Write and cleanup paths keep their own context - a cancelled request must not
abandon a flush half done.
* helm: label the hook Job pods so selectors can reach them
The bucket hook pod carries only managed-by/instance and the volume
resize hook pod carries no labels at all, so nothing keyed on the
standard app.kubernetes.io set can address either of them. Give both
the same name/chart/managed-by/instance/component labels the other
workloads use, with the component naming each hook.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: assert the whole label set on the Job and its pod
The block checked three keys and only looked at the pod's component, so a
missing chart/managed-by label, a wrong component on the Job, or a Job and
pod that disagree would all have passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: compare the hook's release labels against a real workload
Presence alone let a wrong value through. The name/instance/chart values now
have to match a workload that already renders them, which also keeps the
chart version out of the test. managed-by stays a presence check: the chart
puts it on workload metadata but not on pod templates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: quote the resize hook's managed-by and instance labels
Matches the bucket hook, and keeps a numeric release name a string instead
of an int the API server rejects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Short-lived clusters report once under a fresh raft topology id and never
again, so CI runs and demo stacks each become their own cluster. Held
forward for the whole active window they pile up, and the volume server
line climbs every day while capacity stays flat.
Sum the fleet series over confirmed clusters only, the same set the
version and OS charts already use.
The reports are what tell us which versions and cluster sizes are
actually in use, and almost nobody flips the flag on, so the numbers we
have are close to useless. Default it on for master, server and mini,
and say in the flag help and the startup log how to turn it off.
Nothing new is collected: still an in-memory cluster id that changes on
restart, version, os, server counts, volume count and disk bytes, sent
once a day by the leader master only.
Counting by node lost the per-node ShardsInfo, and with it the shard ids the
old summary printed -- the message an operator gets when the pre-delete check
refuses now says only how many shards each node holds.
That is the wrong half. A set holding shards 0-9 and one holding 4-13 are
both "10 shards", and only the ids say whether what survived can rebuild the
volume, or which node to go looking at. Keep the count and list the ids
beside it.
* ci: move the fusermount3 repair into a composite action
Three copies of the same block were already drifting apart, and the
target comes from PATH: only ever add setuid root to a root-owned,
non-symlink binary under the system bin paths, and say why otherwise.
* test: say that the process exited in the wait errors
"process exit status 1 before ... accepted connections" is missing its
verb. Also mark the SIGTERM return discarded - it fails with
os.ErrProcessDone exactly when the select below already handles it.
* ci: prefer the distro fusermount3 over escalating a shadow copy
The shadowing /usr/local/bin/fusermount3 is not root-owned either, so
setting its setuid bit would have handed root to a binary the runner
user owns - the repair now symlinks the distro one earlier in PATH and
touches nothing, keeping the in-place chmod for a root-owned binary with
no distro alternative. A setuid bit only grants root when root owns the
file, so accept an existing one only then.
* ci: run the FUSE workflows when the shared action changes
Their paths filters listed each workflow file but not the composite
action all three now call.
generateEcShards writes shards beside the source volume, so encoding a
volume that lives on a non-default medium puts them on that medium while
-diskType still says hdd. The pre-delete check counted only the -diskType
bucket, so it saw a complete set as zero shards, called it unrecoverable
and aborted -- leaving the volume as both a .dat and a full shard set,
which every later reader then disagrees about.
Count by node across disks, as waitForEcShardsToRegister in the same file
already does. The spread check is unaffected: it locates shards through
collectEcShardBitsByNode and only uses diskType to find free slots.
* ci: restore the setuid bit on a shadowed fusermount3
Newer ubuntu-22.04 runner images carry a source-built fusermount3 in
/usr/local/bin that shadows the distro one in PATH and is not setuid
root. go-fuse looks the helper up through PATH, so every unprivileged
mount fails with "mount failed: Operation not permitted".
* test: fail a fuse test as soon as its mount process dies
A mount that cannot mount at all exits within a second, but the harness
still waited out the 30s readiness timeout and then reported "mount
point not ready within timeout", leaving the real cause buried in the
log tail. Watch the child processes and report their exit instead.
* mount: report a failed mount without a goroutine dump
A mount failure is an environment problem - no /dev/fuse, fusermount not
setuid, stale mount point - and the all-goroutine stack dump Fatalf adds
buries the one line that says so.
* s3: register the advertised ip with the master
The cluster address came from the bind ip, falling back to the
auto-detected interface, so -ip never reached the S3 registration.
weed mini -ip=localhost binds the wildcard and ended up registering
whatever interface happened to sort first -- on a host with VPN
interfaces, an address that stops routing once the tunnel drops.
IAM changes are pushed to registered S3 servers over gRPC, so every
mutation then blocked the full 10s propagation deadline before logging
a failure, and cluster.ps and the admin UI listed a node nothing could
reach. Identities still arrived through the /etc/iam metadata
subscription, so this cost latency and visibility, not credentials.
Add an advertise ip to the gateway option, preferring it over the bind
address, and wire the parent -ip through server, filer and mini.
* s3: treat any unspecified bind address as a wildcard
net.ParseIP + IsUnspecified covers ::, [::] and the expanded IPv6 forms
instead of only the 0.0.0.0 literal, so an IPv6 wildcard bind no longer
registers an address peers cannot dial. Host names parse as nil and stay
addresses in their own right. Apply the same guard to the advertised ip.
* s3api: load document-style policies from the advanced IAM config
The advanced IAM file doubles as the S3 identity config when only
-s3.iam.config is given. protojson drops its "document" field, so every
policy landed with empty content and warned "skipping invalid policy" on
each reload. Worse, if the same file also declares identities the empty
content sticks in the policy map and fails the whole runtime policy sync
into the IAM manager, so policies created later never reach it.
* iam: skip an unparsable policy instead of failing the whole runtime sync
One policy the engine cannot parse aborted SyncRuntimePolicies before it
touched anything, so every other policy stayed unsynced and the engine
kept serving whatever it last held.
* s3api: reject a non-role RoleArn in AssumeRole as a bad request
arn:aws:iam:::user/name can never resolve to a role, but the handler ran
it through the trust-policy check and answered "not authorized to assume
role", pointing the caller at a permission problem they do not have.
* s3api: build the policy content before touching the entry
Deleting "document" up front meant a marshal failure left the policy with
neither field, so a later rewrite would emit it with no definition at all.
* iam: pin the fail-closed handling of an unparsable policy
Say in the comment that dropping it from the desired set deletes it from
the engine on purpose, and cover it with a test.
* s3api: widen the non-role RoleArn test to canonical ARN shapes
The reported ARN omits the account id; a user ARN that carries one, and a
non-principal ARN, must be rejected the same way.
* volume: recover .idx rows overwritten by tiered deletes
A delete on a read-only volume backed by a remote tier used to write its
tombstone row at .idx offset 0 rather than appending it, so each delete
overwrote one more row at the front and lost the Put rows indexing the
first needles in .dat. Those needles 404 even though .dat still holds
them, and rebuilding .idx with weed fix means stopping the server and
pulling the whole .dat back from the tier.
The damage has a fingerprint -- .idx opening with a run of offset-0
tombstones, which a healthy .idx never does -- and .idx and .dat grow in
lockstep, so the lost rows indexed exactly the first N .dat records.
Detect it at load and re-derive them from a header-only walk over the
head of .dat, cheap even against a remote tier, appending only the keys
the .idx no longer names.
* rust volume: mirror the .idx head tombstone recovery
Port the Go detection and repair: an .idx opening with a run of offset-0
tombstones lost the Put rows indexing the first needles in .dat, so
re-derive them at load from a header-only walk over the head of .dat and
append the keys the .idx no longer names.
* volume: put recovered .idx rows back in front instead of appending
Appending left the offset-0 tombstone run at the head, so every later
load re-walked .idx to the tail to notice the volume was already
recovered, and the rows for the head of .dat sat past the .dat-tail row
-- costing CheckVolumeDataIntegrity its O(1) path and breaking the
ascending append order BinarySearchByAppendAtNs assumes.
Rewrite .idx as the recovered rows followed by its current contents,
through a temp file and a rename. .idx is back in .dat append order, so
a later load stops after reading one row.
* volume: keep the .idx mode when the repair replaces it
The recovery renames a fresh temp file over .idx, so a fixed 0644 (Go)
or whatever the umask allows (Rust) would silently widen an index an
operator had locked down. Carry the mode off the file being replaced.
* s3tables: read Iceberg manifest lists that omit the Avro format version
The Iceberg spec pins the Avro header metadata of manifest files but says
nothing about manifest lists, so writers disagree. Java and PyIceberg record
"format-version"; DuckDB writes no header metadata at all. iceberg-go reads a
missing entry as v1, so every v2 manifest listed in a DuckDB-written list is
rejected with
manifest file's 'format-version' metadata indicates version 2,
but entry from manifest list indicates version 1
and, because v1 has no "content" field, delete manifests silently decode as
data manifests.
ReadManifestList derives the version from the record schema the writer
embedded - v2 added "content" and the sequence numbers, v3 added
"first_row_id" - and splices it into the header before handing the bytes to
iceberg-go. Lists that already carry the entry, and input that is not a
parseable Avro container, go through untouched.
* iceberg: parse DuckDB-written manifest lists in maintenance and data preview
Every manifest list read - the four maintenance operations and the admin
table data preview - went straight to iceberg-go, so tables written by DuckDB
failed detection and all of compact, remove_orphans, rewrite_manifests and
expire_snapshots before they touched anything. Route them through
s3tables.ReadManifestList, which recovers the format version the writer left
out of the Avro header.
This also restores the manifest content type on those tables: with the list
read as v1 every delete manifest looked like a data manifest, which hid
deletes from the compaction guard and made the preview report a table with
position deletes as having none.
* telemetry: total disk usage over time counted each cluster on one day
GetMetrics aggregated s.instances, which holds only each cluster's most
recent report. Every cluster therefore landed in a single date bucket --
the day it last reported on -- so the chart plotted the disk usage of
clusters that went silent that day, and piled the whole live fleet onto
today. Aggregate the daily histories instead, reusing the day alignment
that the per-cluster size series already does.
* telemetry: don't pad the charts with days the server has no history for
The dashboard asks for 30 days, but daily history only starts when a
server first collects it, so the charts opened on a run of zeros and then
jumped -- reading as a fleet that appeared overnight. Start the window at
the oldest sample on hand when it is younger than the requested range.
* s3api/iceberg: report the reason a table schema was rejected
newTableMetadata swallowed the iceberg-go error and returned nil, so every
schema the metadata builder refused came back as a bare 500 "Failed to build
table metadata". A v3-only column type is the common case: creating a table
with a variant field but no format-version 3 property leaves the client with
nothing, while "variant is not supported until v3" sits in the server log.
Return the error instead and classify it. Schema, spec and argument failures
are the caller's input, so they answer 400 with the underlying reason; the
rest stay 500. Paths that build placeholder metadata with no schema keep
their existing 500 via newEmptyTableMetadata.
* s3api/iceberg: fail LoadTable when placeholder metadata cannot be built
buildLoadTableResult dropped a nil from the placeholder path straight into
the response. That serializes as "metadata":null under HTTP 200, which no
Iceberg client can parse -- a worse outcome than the 500 the nil was meant
to signal.
Return an error instead and let the five callers answer 500. The nil-return
convention goes away with it, so the commit and transaction paths check an
error rather than a sentinel.
* s3api/iceberg: route rejected schemas through writeManagerError
The two helpers added here duplicated work the package already does.
writeManagerError is the canonical error-to-response mapper -- it already
downgrades client-input failures to 400 and defaults the rest to 500 -- so
teach it the iceberg-go schema and spec sentinels instead of standing up a
parallel classifier. The placeholder wrapper was a pure alias for
newTableMetadata with nil arguments; call that directly.
No behavior change beyond the 500 message, which now reads err.Error()
like every other manager error rather than carrying its own prefix.
curl (unlike wget --spider) exits 0 as soon as it gets any HTTP response, even 401/403 - required if the filer rejects unauthenticated requests when JWT auth is enabled
The staged-new-volume placement skipped a disk holding the vid's EC shards using only the in-memory ecVolumes map, missing a shard present on disk but not mounted. Also scan the candidate disk for <vid>.ecNN files, so the promise holds regardless of mount state.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
volume: skip a shard-holding disk when staging a decoded volume
ReceiveFile staged-new-volume mode picked any free disk of the target
medium. Skip a disk that already holds the vid's EC shards (Go
DiskLocation.FindEcVolume / Rust ec_volumes), so a decoded .dat never
lands in the same directory as a shard. This lets a caller safely stage
onto a shard host that has a spare disk, instead of requiring a host with
no shard of the vid at all.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
Decoding EC shards back to a normal volume in place reconstructs <vid>.dat
in the shards' own directory, so the vid is momentarily registered as both
an EC and a normal volume in one location — the load/scan path then sees it
as both, risking mount ambiguity and needle loss. VolumeEcShardsToVolume
still supports that in-place path; this adds the primitives to decode onto
a *clean* peer instead:
- ReceiveFile gains a staged-new-volume mode: when the volume does not
exist here and ReceiveFileInfo.disk_type is set, pick a free-slot disk
of that medium and write <base><ext>.copying (not a valid volume name,
so the scanner never half-loads a partial push).
- VolumeEcShardsToVolume gains from_staged: adopt the pushed .dat/.idx/
.vif — rename .copying into place under a .note in-progress marker,
then mount — so <vid> lands on the peer only as a normal volume.
The caller decodes the shards off-box and streams the finished volume to a
peer holding no shard of the vid on the target medium. Go and Rust volume
servers get identical handlers. Proto: ReceiveFileInfo.disk_type (12; 8-11
reserved for versioned-EC), VolumeEcShardsToVolumeRequest.from_staged (3) +
disk_type (4).
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
VolumeMarkReadonly mutates raft-replicated master topology, so it must
reach the leader. notify_master_volume_readonly targeted the static seed
(config.masters.first()), so after any master failover it hit a follower
and failed "not current leader". Prefer current_master_url (the live
leader the heartbeat tracks), fall back to the seed before the first
heartbeat, mirroring store_ec.rs and Go's vs.GetMaster().
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* s3: list the buckets an attached IAM policy grants
ListBuckets served an identity authorized by an attached IAM policy only
the buckets it had created itself. A user granted s3:ListBucket on a
bucket someone else provisioned could GetObject and ListObjectsV2 against
it, but the bucket never showed up in the listing any S3 client uses to
build its bucket picker.
The owner-index fast path is only valid for an identity whose grants name
every bucket it can reach, and the routing check assumed a policy could
never be enumerated. Read the names out of the policy instead: statements
that allow s3:ListBucket on a concrete bucket ARN become candidates, and
the per-bucket permission re-check still decides what is listed. A policy
that can reach a bucket it does not name -- a wildcard resource, a policy
variable, a NotResource, an STS session policy -- falls back to the full
scan, which evaluates the policy per bucket.
* s3: share the attached policy name lookup
authorizeWithIAM and the ListBuckets enumeration both built an identity's
policy names the same way, its own plus the ones from its enabled groups.
Pull that into one helper so group eligibility is decided in a single place.
* s3: read policy actions the way the IAM authorizer matches them
The IAM authorizer matches action names case-insensitively, so a policy
granting "S3:LISTBUCKET" or "S3:*" authorizes a list. The ListBuckets
classifier read those actions with the local case-sensitive matcher and
found no grant, so once the owner index was ready the buckets that policy
allows dropped out of the listing.
Match the action the looser way in the classifier: case-insensitive, and
true for any pattern holding a policy variable. Over-matching only costs a
candidate the per-bucket permission check then rejects, while under-matching
hides a bucket the caller can read.
* s3: infer a multipart grant in any case
The classifier matches action patterns case-insensitively but looked the
requested action up in a canonical-cased set, so "S3:UPLOADPART" missed the
s3:PutObject inference that the authorizer makes. Key the set for lookup in
lower case, matching how the IAM authorizer holds it.
* azure: let the blob endpoint be configured
The service url was always derived as <account>.blob.core.windows.net,
which leaves out Azure Government, Azure China, and private endpoints.
Name the blob service url instead and those accounts become reachable.
The url has to be https, since the account key or the bearer token would
otherwise travel in the clear.
* azure: reject an endpoint that carries no hostname
A url like https://:443/ has a host of ":443", so the emptiness check on
Host let it through and the request only failed once it reached Azure.
The hostname is what has to be there.
df on a mount shows the space the cluster gives up to the data: every
replica of a regular volume, every shard of an ec one. That is the honest
answer for capacity planning, but it is not the question a user asks when
they want to know how much of their data is stored.
Add -df.logical. The master reports the logical sizes alongside the raw
ones: one replica per regular volume, the data shards of each ec volume
counted once. Free space is divided by the copies the requested
replication makes, so used plus available stays the amount of data the
mount can still write, and it comes off the cluster-wide usage rather
than one collection's, since capacity is cluster-wide too.
Statistics through a filer resolves an unset replication to the filer's
default rather than the master's, matching where the writes it is sizing
for actually land.
The flag governs the quota check too, so a mount has one notion of how
much it is using. A filer that predates the new fields sends zeros, and
the mount keeps reporting the raw sizes.
Statistics aggregates the volume layouts of a collection, but EC volumes
are tracked outside collectionMap, so they were reported as nothing. A
mount over a cluster whose volumes have mostly been encoded showed a df
used size of a few GiB against terabytes of EC data.
Walk the data nodes and add the EC volumes of the requested collection.
Every shard copy counts, parity included, the way a regular volume's used
size counts every replica, so used size stays the space the cluster
actually occupies.
File count comes from the volume-wide .ecx and .ecj counts, taking the
largest a holder reports rather than summing them: both files travel with
the shards on a move, so several nodes can report the same tombstones.
* azure: authenticate the blob sink with Entra ID
Shared account keys have to be distributed and rotated everywhere a sink
runs. Leaving account_key empty now falls back to the identity chain, so a
workload identity or managed identity carries the authorization instead.
* azure: authenticate remote storage with Entra ID
The remote storage client demanded an account key and refused to start
without one. Fall back to the identity chain when it is absent, and let
azure.client_id pin a user-assigned identity.
* azure: reject a malformed storage account name
The account name is interpolated into the service URL, so a name carrying
a "/", "?" or "@" moves the authority elsewhere and an authenticated
request follows it. Hold callers to Azure's own naming rule instead.
* azure: keep a leftover environment key off the identity path
A configured client id asks for Entra ID, but AZURE_STORAGE_ACCESS_KEY
still filled in the account key behind it. An old mounted secret would go
on authenticating until it rotated, and the failure then blamed the key.
* azure: say what the identity path reads from the environment
A pinned client id alone is not enough for workload identity: the tenant
and the projected token come from the environment, and missing them only
surfaces later, when a token is first requested.
* s3: apply filer identity changes despite a static config file
A -config file with inline identities disabled the metadata-subscription
reload entirely, leaving the best-effort filer->s3 push as the only way
s3.configure changes could reach a running gateway. Reload on IAM events
regardless: the merge keeps the file's identities protected, and a full
credential-manager snapshot now also drops dynamic identities the store
no longer has, so revocation works without a restart.
* s3: log identity propagation failures as warnings
* s3: retry failed IAM reloads and reconcile policies and groups
An event-driven reload that fails now hands off to a coalescing retry
loop, so a transient filer error cannot strand a revoked credential
until the next IAM event. Full-state merges also drop dynamic policies
the store no longer has, keeping the static file's, and treat the group
snapshot as authoritative even when empty.
* s3: serialize IAM configuration loads
The SIGHUP file reload, subscription reloads, the retry loop, and the
postgres poll run on different goroutines. Without an end-to-end lock a
load holding an older store snapshot can commit after a newer one and
revert it. Hold reloadMu from snapshot through commit in both load
entry points; partial merges from pushed updates stay lock-free and
self-heal through the next event-driven reload.
* s3: keep static-file groups through full-state reconciliation
Group names from the static config file are tracked like identities and
policies, and a full snapshot that does not carry them keeps the current
definition and its memberships instead of dropping them.
* credential: include groups in postgres configuration snapshots
Full-state reconciliation treats absent groups as deleted, so a
snapshot that never carries them would erase every dynamic group.
* s3: revoke static-file groups dropped from the config file
A file reload is authoritative for the file's group set while keeping
dynamic groups, mirroring how full snapshots are authoritative for
dynamic groups while keeping the file's.
* credential: fail filer snapshots on unreadable entries
A skipped identity or policy file made the load report success with an
incomplete snapshot, which reconciliation reads as deletion and the
retry loop never sees. Unparseable content is still skipped: it is
durable, matches boot behavior, and must not block reloads forever.
* s3: ignore groups in static config files
Groups are managed through the IAM API and the dynamic store; no
deployment defines them in a bootstrap config file. Ignoring them with
a warning removes the two-directional group merge: full snapshots are
plainly authoritative and file reloads never touch groups.
* s3: require a bucket-policy action to write a bucket policy
PutBucketPolicy and DeleteBucketPolicy were gated on ACTION_WRITE, the same
action that grants object writes. An explicit Allow in a bucket policy
short-circuits IAM entirely -- authRequestWithAuthType sets policyAllows and
skips VerifyActionPermission -- so anyone who could write an object could
author a policy granting itself, or anonymous, anything on the bucket.
That is what separates a bucket policy from the sibling bucket controls also
gated on ACTION_WRITE: rewriting cors or lifecycle can destroy data, but only
a policy hands out access.
Give the two verbs their own actions, mapped to the AWS names that were
already defined but unrouted. ACTION_ADMIN would also have closed it, but it
resolves to s3:* for IAM identities, forcing a blanket grant on a user holding
a precise s3:PutBucketPolicy. Admins are unaffected, since isAdmin
short-circuits CanDo, and an operator can delegate with PutBucketPolicy:bucket.
The route binding is asserted from the router source: checking the action
constants alone still passes when the route says ACTION_WRITE.
* s3: also read the action from a direct iam.Auth call in the route test
Routes read iam.Auth(cb.Limit(handler, ACTION)), a multi-value pass-through:
Limit returns (http.HandlerFunc, Action) and those become Auth's parameters, so
the action Auth authorizes on is Limit's second argument and the two cannot
disagree -- Auth(Limit(h, X), Y) does not compile.
A route that skipped Limit and called Auth with its own action would compile,
though, and the test reported that as a missing route rather than as the wrong
action. Recognise the two-argument Auth form so it names the action instead.
* s3: make the bucket-policy actions grantable through an IAM policy
The new actions close the escalation only if an operator can grant them, and
they were not reachable: MapToStatementAction had no entry for PutBucketPolicy,
so an IAM policy naming s3:PutBucketPolicy was rejected outright with "not a
valid action". GetBucketPolicy was unmapped the same way.
DeleteBucketPolicy was mapped, but to ACTION_ADMIN -- granting an identity
permission to delete a bucket policy handed it full administrative access.
Map all three to the actions the router now uses, and add the reverse
direction so an identity holding them renders back as a policy statement
instead of a bare "s3:".
* admin: offer the bucket-policy permissions in the user editor
The two new actions are otherwise only grantable by hand-editing identity JSON
or by calling the IAM API, so an operator using the UI cannot delegate bucket
policy management without granting Admin.
Regenerating this file also picks up codegen the repo has not taken yet: the
checked-in _templ.go files were produced by templ v0.3.1001 while go.mod pins
v0.3.1020, so the generator rewrites the attribute-value calls. That churn is
confined to this one file; running `make generate` in weed/admin reproduces it
across all 36.
* volume: gate the admin RPCs that only shell and workers call
checkGrpcAdminAuth covered 19 of the 48 VolumeServer RPCs, so an operator who
sets -whiteList expecting it to cover the gRPC surface gets partial coverage.
Extend it to ten that mutate state and are only ever called by the shell or a
worker: SetState, VolumeCopy, the EC generate/rebuild/copy/unmount/to-volume
pair, both tier moves, and VolumeTailReceiver. That is safe because the same
callers already reach gated RPCs today -- VolumeMarkReadonly, VacuumVolume*,
VolumeEcShardsDelete, VolumeDelete -- so a whitelist deployment already lists
those hosts. Nothing here is on a master or peer path, which is what made the
earlier fail-closed gate break multi-host clusters.
The split is by caller rather than by blast radius: the guard matches a peer IP
against the whitelist, and a whitelist holds masters, shell hosts and workers,
not every peer volume server. Gating a call one volume server makes to another
would break replication, EC and tiering, so those stay open.
Two test fakes embedded a nil grpc.ServerStream and only implemented Send;
they now implement Context, which the streaming RPCs read to authorize.
* volume: fail the build when a gRPC method skips the admin gate
The admin gate is an opt-in list in a 48-method service, which is how it
drifted down to covering 19 of them: nothing tied adding an RPC to deciding
whether it needed the gate.
Parse volume_server.proto, walk the AST of every *VolumeServer method, and
require each RPC to either call checkGrpcAdminAuth or appear in
ungatedVolumeServerRPCs with the reason it stays open. A stale entry naming an
RPC that no longer exists fails too, so the list can't quietly stop exempting
anything.
The exemptions are the cluster-internal calls -- replica sync, EC shard
distribution, vacuum reads, backup, tailing -- plus the read-only and liveness
RPCs. Closing the cluster-internal ones needs a peer identity rather than an
IP whitelist; recording them here makes that a visible decision instead of an
omission.
The AST walk also corrects the count: a line-window scan credits
VacuumVolumeCheck and VolumeServerStatus with a neighbouring function's guard.
* volume: fix EC decode/reconstruct index locality under -dir.idx
EC->replicated decode failed under -dir.idx and on multi-disk with "volume not
found on disk". The reconstruct rebuilds the .dat on the data disk but the
on-demand VolumeMount scans only the data directory, matching on .idx/.vif;
with the rebuilt .idx off in the index directory it matched the volume's
leftover EC .vif and skipped the volume as EC metadata.
- Resolve the EC .ecx local-first: prefer the copy co-located with the shards
over the shared -dir.idx copy, with a non-empty preference so a 0-byte local
stub still yields to a valid sibling (the cross-disk fallback).
- Co-locate the rebuilt .idx with the .dat at the end of the reconstruct so the
mount finds it; sweep .ecx/.ecj from both the data and index directories on
Destroy so a stale copy cannot re-mount as a phantom EC volume.
- Add VolumeConsolidateIndex: once the EC shards are deleted, unmount, move the
.idx/.sdx from the data disk back to the -dir.idx directory (copy fallback
across filesystems), and remount. A no-op without -dir.idx.
* volume: tests for EC index locality (local-first .ecx, sweep, consolidate)
- NewEcVolume prefers a non-empty local .ecx over the shared index dir, and a
0-byte local stub yields to a non-empty shared copy (the #9212 fallback).
- Destroy sweeps .ecx/.ecj from both the data and index directories.
- ConsolidateVolumeIndex moves a co-located index back to the -dir.idx dir and
keeps the volume mounted; no-op without a separate index dir.
- RenameOrCopyFile moves a file and drops the source.
* volume: relocate the decoded index in place, without a read gap
ConsolidateVolumeIndex previously unmounted the volume, moved the index, and
remounted it. Between the EC-shard delete and the remount the volume had neither
a normal nor an EC form mounted, so a read landing in that window got a
not-found (or was proxied away).
Move the index in place instead: RelocateIndexTo takes the data-file write lock,
closes the needle map and data backend, moves the .idx (and derived .sdx), then
retargets dirIdx and reloads — the same close-swap-load CommitCompact uses. The
volume never leaves the mounted set, so a concurrent read blocks briefly on the
lock rather than failing. The test now writes a needle before consolidating and
reads it back after, proving the in-place reload keeps the volume serving.
* volume: address review — maintenance guard, no orphan on copy failure
- VolumeConsolidateIndex now rejects the request under maintenance mode, like
VolumeConfigure and the other mutating volume RPCs.
- RenameOrCopyFile rolls the cross-device copy back when the source cannot be
removed, so a failed move never leaves two divergent copies (the loader would
keep the data-dir one while the idx-dir orphan goes stale).
- RelocateIndexTo logs a failed reopen-after-failed-move instead of swallowing
it, since that leaves the volume unusable until the next load.
The sentinel stores hardcoded a 30s read timeout and a 1m retry backoff.
After a sentinel failover every request that picked a pooled connection to
the old master sat there for 30s before the connection was retired, and the
pool timeout derived from it (read timeout + 1s) queued the rest behind
them. The other redis stores took the go-redis defaults with no way to tune
anything.
Read the dial, timeout and pool knobs from each redis store section instead,
keeping the go-redis default for every key left unset.
security.GetJwt reads the "jwt" query parameter before the Authorization
header, and the proxy forwarded the caller's whole query apart from
proxyChunkId. So on a read, where the filer mints a volume token and sets the
header itself, a caller-supplied ?jwt= silently outranked it: the volume
server validated a credential the caller chose rather than the one the filer
attached, and the read failed with a 401 the filer could not explain.
Drop it on the read path, where the filer owns the credential. Writes keep
theirs -- the proxy forwards a writer's own AssignVolume token either way, so
the query parameter is just a second channel for the same credential and
stripping it would break a caller that presents it that way.
Nothing in the tree passes a jwt by query; maybeAddAuth always sets the
header.
The S3 write path cut fixed 8MB chunks, so an object stored through S3
chunked differently from the same bytes stored through the filer, WebDAV
or a mount, and -maxMB had no effect on it. Read maxMB from the filer
configuration at startup and use it, falling back to 8MB when the filer
reports none.
* filer: add a placement overlay seam for the write path
New volumes take their disk type, replication, and data center from the
explicit request or the matched filer.conf rule. That leaves no way for a
feature to steer a whole collection onto a medium without an operator
writing an fs.configure rule by hand.
Add a generic PlacementOverlay hook on the filer: a func that maps a
collection to a placement override, installed by a factory the way the
plugin-worker handlers register. detectStorageOption consults it between
the explicit request value and the filer.conf rule, so it overrides the
rule but yields to a value the caller asked for.
The seam names no feature concepts, so it stays generic; a downstream
build registers the overlay it wants (e.g. a storage-class Landing tier).
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* filer: address review on the placement overlay seam
Honor ResolvePlacement's ok flag explicitly rather than relying on empty
values falling through the util.Nvl chain, and log at V(4) when the
overlay steers a collection. Document that RegisterPlacementOverlay is
init-only, so the unsynchronized read in NewFiler cannot race the write.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* filer: claim the base fid when minting a volume read token
GenJwtForVolumeServer stamped the fid verbatim, but the volume server strips a
trailing _N delta suffix before comparing the claim, so a token minted for a
batch-assigned fid like 3,01637037d6_1 was checked against 3,01637037d6 and
never matched. Reading such a chunk through the filer returned 401 wherever
jwt.signing.read.key was configured.
Strip the suffix before minting, via a helper shared with the proxyChunkId
validation that was already doing the same thing inline.
* filer: don't mint a volume write token for an anonymous proxy caller
The ?proxyChunkId= branch dispatches and returns before the JWT gate, so
whatever credential the proxy attaches is reachable without authentication.
It attached a token from maybeGetVolumeReadJwtAuthorizationToken, which fell
back to the write signing key when jwt.signing.read.key was unset -- the
configuration scaffold/security.toml recommends for a filer, since read JWTs
are only supported in a master+volume setup. An anonymous
DELETE /?proxyChunkId=<fid> therefore arrived at the volume server holding a
write-key token scoped to that fid, and the volume server honored it.
Sign read tokens with the read key only. The fallback bought nothing on a
read anyway: a volume server enforces read JWTs solely when that same key is
set, so when the fallback fired the read was unchecked regardless.
Mint only for reads. Writers proxied through the filer carry their own volume
JWT from AssignVolume, forwarded with the rest of the caller's headers, so
weed mount -filerProxy uploads are unaffected. Moving the dispatch below the
JWT gate instead would have broken them, since that token is signed with
jwt.signing rather than jwt.filer_signing.
On a read with nothing to mint, drop the caller's Authorization rather than
relaying it: there it is a filer credential, and forwarding it would hand a
volume server a token it never used to see.
* filer: keep proxied writes out of the read concurrency semaphore
The semaphore is named and documented for reads -- it exists so replication
bursts can't open hundreds of connections to one volume server -- but it was
applied to every proxied method. A write queued behind sixteen in-flight reads
can wait past the 10s default expiry of the AssignVolume token it carries, and
the volume server then answers 401. shouldReassignUpload treats a 4xx as final,
so the uploader replays the same expired token instead of re-assigning and the
write fails up to the caller.
This only became reachable once the filer stopped re-minting a fresh token
after the wait.
LookupFileId only requires the fid to contain a single comma, and the value
is pasted straight into the volume server URL path, so ?proxyChunkId=3,x/../../status
resolves to volume 3 and then addresses an endpoint the caller never named:
Go sends the dot segments verbatim, the volume server's mux cleans the path
and redirects to /status, and the filer follows the redirect and relays the
body. That reaches any handler on the volume server -- status, stats, the UI --
past a filer that operators expect to be the only exposed surface.
Parse the fid before the lookup and answer 400 when it doesn't parse.
A trailing _N delta suffix from batch assigns is legal, so it is stripped
first, but only when it is a non-empty run of digits. The volume server strips
at the last "_" unconditionally, which is safe there because its fid came out
of a path the mux parsed and so cannot hold a "/"; here the value is raw query
input, and an unguarded strip would reduce "3,01637037d6_1/../../status" to a
valid fid and wave the traversal through.
The queue holds sixteen sealed windows, which is a memory bound only
while a window is BufferSize. An entry larger than that grows its window
to fit, and the depth then multiplies straight through: sixteen queued
copies of a 100 MB window is 1.6 GB of flush data alone.
Account the queued bytes and make producers wait once they pass the
ceiling the depth was chosen for. What is charged is the pooled slab
rather than the window length, since mem.Allocate rounds up to a size
class and the queue holds the whole slab. A window larger than the whole
budget still goes through on its own, so an oversized entry is never
stuck.
Windows are admitted in the order they were sealed. A producer can now
park here for seconds, and letting a later window overtake an earlier one
would persist them out of order and walk lastFlushedOffset and
lastFlushTsNs backwards.
A window is copied into its slab under the write lock, before the
reservation is taken, so a burst of concurrent oversized writers would
each hold a full copy in hand while queueing up -- memory the budget
never sees. Large writers wait for queue headroom before they take the
lock, which throttles the burst; it does not bound it, since a writer
that passes the check still seals unconditionally.
Take any room in the queue before the shutdown escape, too: the window is
already sealed by then, so dropping it loses records the caller was told
were accepted. A shutdown that races a full queue can still drop one --
that predates this change and needs the flush loop's lifetime reworked.
Size a grown window to the entry rather than to twice it: the extra room
only bought space for a second oversized record in the same window, which
doubles the flush copy and the snapshot taken of it. The overflow guard
halved its bound for that doubled allocation, so raise it to match what
is now allocated and what maxBufferSize documents.
* volume: reject needle blob writes to read-only volumes
WriteNeedleBlob appends the blob to .dat and only then calls nm.Put. On a
read-only volume the needle map is a SortedFileNeedleMap whose Put always
fails, so the append is never indexed and never rolled back.
Nothing upstream stops this: volume.check.disk picks its targets from the
master's cached topology, which goes stale the moment a volume server marks
a replica read-only itself — a failed data integrity check at load, or an
EIO quarantine. Each sync attempt then grows the .dat of a replica that is
supposed to be frozen by one unindexed needle, and reports it as "invalid
argument", the bare os.ErrInvalid the needle map returns.
Check IsReadOnly before touching .dat, same as the upload path does.
* volume: say which needle and volume failed to index
An index write that fails surfaced as a bare errno with no volume, no needle
and no file — "invalid argument" for a read-only needle map, or a plain
ENOSPC when .idx lives on its own filesystem via -dir.idx. Both were logged
at V(4), so by default the operator saw only the errno the client got back.
* filer: write the metadata log in pieces a volume server will accept
A single oversized metadata event grows the log buffer past the volume
server's fileSizeLimitMB, and the flush of that buffer is then rejected
forever: the retry loop has no exit, so the blob at the head of the queue
blocks every later flush and the metadata feed stalls until restart.
Split the flushed buffer into BufferSize pieces, on record boundaries
where possible so each piece still decodes on its own, and retry each
piece separately so a partial success is not replayed. Log files are
already read as a chunk stream, with a whole-file fallback when a chunk
does not decode standalone, so a record may cross a piece boundary.
* log_buffer: let go of a window array grown for an oversized entry
An entry larger than BufferSize grows the window array to 2*size+4, and
window arrays cycle through SealBuffer rather than being freed. One such
entry therefore leaves every later window carrying its size, and
currentSnapshotView allocates a snapshot as wide as the array on each
window, so a few KB of metadata keeps paying for it.
Drop the array when SealBuffer hands it back. Growth is on demand, so
the next oversized entry just reallocates.
* iceberg maintenance: store merged data files as chunks, not inline
saveFilerFile had no size threshold, so compaction wrote whole merged
parquet files -- hundreds of MB -- as Entry.Content. That puts the
parquet bytes verbatim in the filer store and sends them through the
metadata change log again as one event.
Keep manifests and metadata JSON inline, upload anything larger to
volume servers in chunks, assigning through the filer so the path's
storage rules apply.
* filer: follow the file size limit the volume servers report
The starting piece size is a constant, so a cluster whose
-fileSizeLimitMB is set below it would reject every piece and wedge just
the same. The rejection names the limit, so take it from there and
re-cut the rest of the flush to fit.
Piece the buffer one at a time rather than up front, since the size can
change partway through a flush.
plugin: stop the scheduler lock test racing its own background loops
TestRunLaneSchedulerIterationLockBehavior constructed the plugin with a
cluster-context provider, which makes New start a background scheduler
loop per lane. Those loops call runLaneSchedulerIteration on the same
lane the test then drives by hand, so a loop could consume the due job —
running detection and pushing the next-detection time forward — before
the manual call observed the lock. The Default case then saw the lock
acquired zero times and failed intermittently.
Construct without the provider so no loops start, and set the provider
afterward so the manual iteration can still detect. This is the pattern
scheduler_status_test.go already uses for the same reason.
Reproduced under -race -count=100 -p 4 before, green after.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* util: match transient error messages case-insensitively, expose the message form
The same condition reaches different layers capitalized differently -- a
volume server relays its idle timeout as "I/O timeout" inside a JSON string --
and the callers that grew their own substring lists all lower-case first.
Also split out IsTransientErrorMessage for the paths that carry only the text,
such as the per-file status strings in a batch delete response, and pick up
"no route to host" and "network is unreachable" from the gRPC classifier.
* filersink: classify transient network errors through util.IsTransientError
The local list caught i/o timeout, connection reset, and broken pipe but not
connection refused, no such host, unexpected EOF, the syscall errnos, or the
gRPC and S3 overload codes. Keep only the bare io.EOF case, which is transient
here -- a truncated chunk read -- but a clean stream end elsewhere.
* filer deletion: reuse util.IsTransientErrorMessage for the network patterns
Six of the sixteen patterns were already covered. Keep the ones specific to
this pipeline -- read-only volumes, lookup failures, backpressure -- and note
why context cancellation stays retryable here: it decides whether to requeue
the deletion, not whether to retry a call.
* wdclient: fold the shared classifier into the volume lookup retry check
The string tail duplicated the shared list and missed the syscall errnos and
net.Error timeouts. Keep "connection" and "timeout", which are broader than
the shared classifier on purpose: a volume lookup is a cheap read-only call.
* filer.replicate: commit the kafka offset after replicating, not on receipt
The partition consumer committed the offset as soon as it handed the message
to the channel, so a sink write that failed was logged and the message was
already behind the committed offset -- never redelivered, permanently missing
from the sink.
Commit in onSuccessFn instead, and hold the committed offset behind the
oldest offset that failed to replicate so a restart redelivers from there.
* filer.replicate: delete the sqs message after replicating, not on receipt
ReceiveMessage deleted the message before the replicator had a chance to run,
so a failed sink write dropped it for good. Move the delete into onSuccessFn
and leave the message in the queue otherwise, letting the visibility timeout
redeliver it.
* util: retry transient errors, not just the ones containing "transport"
util.Retry only retried when the error string contained "transport", so a
plain "read: connection reset by peer" from S3 got zero retries. Classify
the error instead: net timeouts, connection resets, and the throttling and
overload replies S3 and gRPC return are all worth another attempt, while a
cancelled or expired context is not.
* filer sync: hold the sync offset behind a failed event
A sync job that returned an error was logged and forgotten, and the
watermark advanced past it anyway. The offset is the durable resume point,
so the event was never replayed: for filer.remote.sync that left the file
present locally, absent on the remote, with no RemoteEntry and nothing to
retry it.
Pin the watermark at the oldest failed event. Later events keep flowing,
but the persisted offset stays behind the failure, so a restart replays it.
The scaffold advertises enable_tls, ca_cert_path, client_cert_path and
client_key_path under redis2, redis2_sentinel and redis_cluster2, but only the
plain redis2 and redis3 stores ever read them, and under a different name,
enable_mtls. Sentinel and cluster setups quietly connected in plaintext.
Build the TLS config in one place and use it from all six stores. enable_mtls
still works. The CA and the client key pair are optional now, so enable_tls
alone verifies against the system roots, and ServerName is left unset so
go-redis validates each address the sentinel and cluster clients dial.
redis2_sentinel and redis3_sentinel stores only passed Username/Password
into redis.FailoverOptions, which authenticates against the Redis
master/replica servers. When Sentinel itself requires auth (requirepass
set in sentinel.conf), go-redis had no credentials to send to it,
causing a NOAUTH error before ever reaching the master.
Add sentinel_username/sentinel_password config options that map to
go-redis's SentinelUsername/SentinelPassword fields, distinct from the
existing master auth credentials.
Deleting the only object under a prefix in a versioned bucket writes a
delete marker and keeps the version history, so the filer directory
survives with nothing a current-version listing would return. A delimited
ListObjects kept reporting that path in CommonPrefixes, because the
prefixes come from the directory tree rather than from the keys, while a
listing scoped inside the prefix correctly came back empty.
Probe a directory before reporting it: one that holds entries but no key
the listing returns is neither a CommonPrefix nor a path the
trailing-slash probe answers for. Empty directories keep the meaning they
have today, and the probe only runs for buckets with versioning
configured, the only ones that can reach this state.
A maintenance snapshot carried only its own labels — merged-files,
delete-groups and friends — and no summary counters, so every engine that
reads a table's size out of the current snapshot summary reported nothing
for it: PyIceberg's inspect.snapshots, Trino's $snapshots and Spark's
DESCRIBE all read total-records, total-data-files and total-files-size
verbatim, and a table lost them the moment compaction touched it.
Accumulate the files each operation adds and removes, and render them the
way the spec defines: the added-*/removed-* counters from the files
themselves, then the totals carried over from the parent snapshot.
Carry a total only when the parent recorded it. Iceberg treats a missing
total as zero, which turns a compaction replacing two files with one into a
negative total-data-files, or a table with millions of rows into
total-records: 0. Leaving the field out lets a reader fall back to the
manifests instead of believing a made-up number.
Compaction also accounts for the delete files it consumes, so a run that
folds every delete into the rewritten data reports them as removed.
The worker assumed every file of a table sits under its catalog path, so
loadFileByIcebergPath stripped the scheme off a recorded location and joined
the remainder onto /buckets/<bucket>/<ns>/<table>. A table the REST catalog
placed elsewhere in the bucket — which is what a client gets whenever the
catalog path is already occupied — then resolves to a doubled path:
lookup /buckets/lake/source/t/lake/source/t-0cd81bca-.../metadata/snap-.avro
so the very first manifest list read fails and the job fails again on every
scan interval, indefinitely.
Resolve absolute references (s3:// URIs and /buckets paths) from the bucket
root and keep relative ones under the table's own directory; the
bucket-relative form is now the canonical key everywhere references are
compared. That directory comes from the metadata location the catalog stores,
so reads, writes and deletes all land where the table's other files are
instead of splitting it across two trees. References outside the table's
bucket are rejected rather than silently misresolved.
Rewritten position-delete files now name their data file by absolute URI,
the way the table itself names it, instead of a path relative to the table.
The dashboard charted one summed disk-usage line, so a step in the total
gave no hint which cluster moved. A new panel stacks each cluster's daily
size as its own band: the top of the stack is the fleet total, each band
is one cluster, and the clusters past the twentieth are summed into an
"other" band so the stack still adds up to the total.
The series is built from the per-cluster daily histories and served by
/api/cluster-sizes. Clusters report roughly once a day at no fixed hour,
so a day with no report carries the previous value forward — dropping it
to zero would sag the total every day as the clusters that have not
reported yet fall out from under it. A cluster that stops reporting past
the active window ends at its last sample instead of holding capacity
forever. Ranking is by the most recent day, tie-broken on cluster id so
the colors do not shuffle between refreshes.
Hover and click resolve to the band under the pointer: Chart.js's builtin
interaction modes match the nearest line, which on a stack of thin bands
is rarely the band being pointed at. Clicking one fills the per-cluster
history lookup below it.
LiveMoveVolume and the copy, tail, delete, mark, replicate, and
configure helpers around it issued every RPC on context.Background(), so
a caller had no way to bound or abort a move once it started. They now
take a context, which the exported LiveMoveVolume in particular needs:
callers outside the shell drive long moves and want to stop them.
The deferred restore in copyVolume runs on a detached, bounded context
rather than the caller's. Marking the source writable again is cleanup,
and cancelling the copy must not skip it and leave the volume readonly —
the same guard balance_task.go already applies for the same reason.
Shell commands pass context.Background(): their Do signature carries no
context, and changing it would touch every command in the package.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* filer: stamp a log position on lookup and remote-cache responses
Metadata events are logged after their store write and stamped with the
filer clock. Reading that clock before serving an entry therefore gives
a timestamp with a causal guarantee: every event at or below it is
reflected in the returned entry. Clients caching filer state can use it
as the entry's version to order the response against subscription
events, including events committed before the call but delivered after
it.
* mount: version open file handles by filer log position
A subscription event refreshing an open handle did a second lookup; a
transient failure left the handle pinned to its old entry with no
retry, since the subscription cursor had already advanced. The deeper
problem is ordering: the handle is a cache written by three unordered
channels — the async invalidation worker, local mutation acks, and
open-time lookups — and overwriting cached state safely requires
knowing which write is newer.
The filer log timestamp is that order, and it now travels with every
value instead of being derived out of band. Events carry it natively;
lookup and remote-cache responses carry the log position stamped before
the serving read; mutation acks carry it in their returned event; and
the local store pairs each read with a version cursor advanced under
the same lock as the store write. Each handle records the version its
entry reflects, and one rule replaces the per-site reasoning: state at
or below the handle's version is old news and must not be installed.
The invalidation itself applies the event's own entry — no lookup, so
no transient-failure window — except under a cached parent, where the
store entry is the ordered merge of the event and anything applied
since, and its version outranks the event's. An uncached parent
receives no store writes, so a hit there would be a stale leftover
masking the event. A vacated path (delete, rename away) keeps the last
entry so unlinked-but-open reads still work. Directory builds version
the completed directory at the listing snapshot and re-invalidate
buffered events at that version, since their mid-build refresh ran
against an incomplete store.
The tests replay every race this replaces machinery for: rollback of a
newer local flush (queued, cached, and read-through), stale leftovers
under uncached parents, the build window including abort, handles
opened after an event was queued, events landing mid-lookup, and
undelivered events at remote-cache time across a filer failover.
* filer: serialize the log position fence with mutations, stamp mutation acks
The fence stamped before an unlocked entry read could precede state the
read returned: a mutation writes storage first and assigns its event
timestamp only at notify time, so a lookup racing that window handed
the mount an entry newer than its fence, and the event's later delivery
looked like fresh news — destroying dirty pages for a change the handle
already had. The mutation handlers already hold an exclusive per-path
lock across read, write, and notify; the lookup and remote-cache reads
now take it shared around the stamp and the read, making the fence
exact: everything at or below it is in the entry, nothing above it is.
A no-change update returns success without an event, leaving the mount
nothing to fence with even though the response confirms current state.
Create and update acks now carry a log position stamped under the same
lock, and the mount falls back to it whenever the ack has no event.
Also regenerate the VT marshalers, which the earlier generation missed:
without them a VT round-trip silently zeroed every log position.
* java: sync filer.proto
* mount: scope store versions to what they vouch for; atomic handle install
The store's version cursor claimed too much. Advanced by local mutation
acks and directory listing snapshots, it inflated the version of store
reads for unrelated paths whose events the subscription still owed, and
those events were then fenced out permanently. The cursor now tracks
subscription progress only — events arrive in log order, so everything
at or below it has been delivered for every path — and a completed
listing records its snapshot as a per-directory floor instead of a
global claim. Local acks never touch it: they version their own handle
directly. Buffered build events advance the cursor at delivery, since
their store write may never happen (abort) while their invalidation is
already queued; their read-through directory pairs no store read with
it, and rename fragments are applied first.
Concurrent first opens raced: a slower opener's older lookup could
overwrite the newer entry a faster opener had installed, while the
monotonic version kept the newer timestamp — an old entry fenced at a
new version, immune to every correcting event. Entry and version are
now installed as one decision under the handle map lock, and an install
that does not outrank the handle's version is dropped.
The remote-cache commit also escaped the fence: it wrote storage and
notified without the path lock, so a lookup's shared-locked fence and
read could land between the two and hand out the cached state
under-versioned. The commit now re-reads and writes under the exclusive
path lock, and backs off entirely when the entry changed during the
download — the concurrent writer supersedes the cached content.
* mount: floors gate store applies; installs respect handle users; renames join the fence
A directory floor certifies the listing state as of its snapshot, but a
delayed event at or below the floor was still applied to the store —
rolling the content back to pre-snapshot state while the floor kept
claiming the snapshot version, so the correcting events were fenced out
of every future read. Events are now gated against the affected
directory's floor, each half of a rename independently.
Fences are lower bounds: a listing or lookup can include a mutation
whose event has not been delivered yet, and that event later passes
every gate carrying state the handle already holds. Such a re-delivery
now advances the version without destroying dirty pages or reinstalling
the entry — invalidating local writes over a no-op was the real damage
in every remaining under-fence window, including the unlocked listing
snapshot, which no per-path lock can serialize.
The concurrent-open install moved from the map lock to the handle lock
every reader, writer, and invalidation synchronizes on, and rejects
what cannot improve the handle: dirty state (local writes would be
lost), unversioned lookup responses (they cannot outrank anything, and
two zero-version opens must not overwrite each other), and anything not
strictly newer. New handles are still fully initialized before the map
exposes them.
Renames committed metadata and emitted events with no path lock, so a
lookup could read the renamed state under a fence preceding its events.
Both rename handlers now hold the source and destination locks, ordered
by path, across commit and notification; descendants of a renamed
directory are not individually locked and rely on the no-op re-delivery
handling above.
* mount: per-entry store versions replace the cursor and directory floors
The store's aggregate versions — a global subscription cursor and
per-directory listing floors — were versions at coarser granularity
than the values they described, and every over-claiming bug in this
series traced to that gap: an aggregate vouching for state its source
never saw. Each store entry now carries the filer log position of the
write that produced it — the event that applied it, or the listing
snapshot that inserted it, recorded in the store's key-value space
under the same lock as the entry write. The store becomes what the
handle already is: a last-writer-wins register with one rule, install
only what outranks the current claim.
The cursor, the floors, their advancement rules, the pairing ordering
constraint, and the floor gating all collapse into that rule. Applies
are gated per entry, each half of a rename independently; an
unversioned local write clears the claim its content no longer proves;
version records lingering after a bulk folder wipe cannot fence a
recreate, since a claim only blocks while its entry exists. Listing
inserts are stamped at build completion, before the buffered replay so
newer replayed events override the stamp.
Filer side, the fence dance every versioned read must perform is now a
single choke point, fencedFindEntry, so a future read RPC gets the
lock-serialized stamp by construction rather than by convention.
* mount: judge no-op re-deliveries against an immutable base, not the live entry
The equal-state skip compared the incoming event to the live handle
entry, but local writes mutate the live entry — size, timestamps,
chunks — so a delayed event re-delivering the base the handle was
opened with no longer matched, and the installer destroyed the dirty
pages and rolled the entry back over nothing new. The handle now keeps
an immutable snapshot of the filer state it last installed or
acknowledged, refreshed at every install and mutation ack (flush acks
snapshot the request entry before the id mapping mutates it), and the
no-op judgment runs against that base: an event carrying the base
brings nothing, whatever the live entry has diverged to since.
* mount: tombstones for versioned deletes, absence floors, copy enrollment
Four gaps in the per-entry version protocol, all the same shape: a
versioned fact with nothing carrying its version.
A deletion is a fact about a path with no entry left to hold it —
clearing the record let a delayed older event resurrect the deleted
path, permanently, since the deletion's own redelivery is
dedup-suppressed. Versioned deletes now leave a tombstone record that
fences without an entry; renames tombstone their source the same way.
Plain records still only block while their entry exists, so records
lingering after a bulk folder wipe cannot fence a recreate.
A completed listing proves absences as well as presences: a name it
omitted was deleted as of the snapshot, and a delayed create below the
snapshot re-creates it. The snapshot is kept per directory strictly as
an absence fence, consulted only when a path has neither an entry nor
a version record — present entries carry their own versions and never
touch it, which is what separates this from the over-claiming floor it
replaces.
A rebuild against a pre-upgrade filer returns no snapshot; stamping
now clears the children's records in that case, so a reinserted entry
cannot reactivate the stale claim its previous incarnation left
behind and reject valid events below it.
Server-side copies installed the copied entry without enrolling in the
base protocol, so the copy's own event differed from the stale
pre-copy base and destroyed writes made to the destination after the
copy. The install now refreshes the base and takes its version from
the fenced readback.
* mount: deletion facts outlive the cache's knowledge of the entry
A versioned delete of a path the store held no entry for recorded
nothing, so a delayed older event recreated the path — permanently,
with the deletion's redelivery dedup-suppressed. The tombstone is now
written whenever a versioned event vacates a path: the deletion is a
fact about the path, not about what this cache happened to hold.
For an absent entry, the listing's absence floor now speaks whatever
older record remains: a tombstone at one position does not exhaust
what is known about the path when a newer snapshot has confirmed the
name still absent, and an event between the two was slipping past
both.
A committed copy whose readback failed installed a synthesized base
with local timestamps; the copy's real event legitimately differs from
it, and was read as foreign state — destroying writes made to the
destination after the copy. The handle now marks that its own event is
en route and adopts that event's state as the base without touching
the live entry or the dirty pages; the adoption is one-shot, so a
genuinely foreign event still invalidates.
* mount: authoritative acks cancel pending event adoption; tombstones scoped and pruned
The copy-event adoption flag could outlive its purpose: a flush after
the failed readback installs a newer base and advances the version, the
copy's own event is then version gated without consuming the flag, and
the next genuinely foreign event was silently adopted — base advanced,
live entry and dirty pages untouched — leaving the mount to later
overwrite that remote change. Every local acknowledgment now installs
its base through one helper that also cancels any pending adoption: the
ack supersedes the mutation the adoption was waiting for.
Tombstones were written for every versioned delete under the mount and
survived directory eviction by design, growing LevelDB with historical
deletions on delete-heavy mounts. They are now scoped to directories
whose cached state the fence actually protects — an uncached parent
never serves from the store nor applies the resurrecting insert — and a
completed listing prunes the direct-child tombstones its absence floor
supersedes, leaving only those above the snapshot. The store gains a
key-prefix visitor for the sweep.
* mount: acked saves install their value; trailer snapshots; direct-child prune range
A version must never advance without its value. saveEntry stamped any
open handle with the acknowledgment's version, but a handle opened
while the save was in flight holds the pre-mutation entry — stamping it
fenced out the events carrying the state it lacked, permanently, with
the local apply performing no invalidation and the redelivery
deduplicated. The acknowledged entry is now installed together with its
version, through the same guarded install the racing-open path uses:
under the handle lock, only when it outranks the handle, never over
dirty local writes.
Empty listings return no in-band snapshot — a snapshot-only response
would be read as an entry by older consumers — so directories that end
empty gained no absence floor and their tombstones were never pruned.
The filer now sends the snapshot in the stream trailer, which older
clients ignore, and the client reads it when no in-band snapshot
arrived. Empty directories get real floors, their tombstones prune,
and their buffered replays gain the snapshot filter instead of the
replay-all fallback.
Version records now encode the parent directory and name separated by
a NUL, making a directory's direct children one contiguous key range:
the tombstone prune scans exactly them under the cache lock, instead
of walking every descendant record — the whole store, for root.
* mount: fix dirty-page loss, uid/gid base, download race, copy adopt, leak; dedup
Correctness fixes from the versioned-invalidation review:
- A foreign delete/rename-away of a file held open with unflushed local
writes destroyed the dirty pages unconditionally. A process may keep
writing to an unlinked-but-open file and those writes were already
acknowledged; preserve the pages when the handle is dirty.
- downloadRemoteEntry stored the handle's base with filer-side uid/gid
while every candidate it is later compared against is in local form,
so under a non-identity UidGidMapper an unchanged re-delivery looked
foreign and force-destroyed dirty pages. Map the base to local.
- downloadRemoteEntry wrote the entry/base/version triple under only the
handle's shared lock, so two concurrent reads of the same remote-only
file could tear it. Serialize the install with a dedicated mutex
(invalidation is already excluded by the exclusive handle lock).
- A committed server-side copy whose readback failed adopted the FIRST
event past the version gate as its base; a foreign write delivered
first was silently swallowed. Adopt only an event whose content
matches the synthesized base — the copy's own event — and install any
other normally.
- The deferred-create path relied on AcquireFileHandle installing the
passed entry on a pre-existing handle, which the version rework
dropped. Restore that install in the compat wrapper; the versioned
open path keeps its gated install.
Growth and hot-path cost:
- Per-entry version records and tombstones leaked when a directory was
evicted or read-through without a rebuild. An uncached directory
gates its own inserts, so its records fence nothing; clear a
directory's child version records when it is wiped for eviction.
- FindEntry paid for the version KvGet on every lookup/getattr cache hit
and threw it away. FindEntry now reads only the entry; the hot
lookupEntry cache-hit path skips the version entirely.
Cleanups:
- Extract ackVersionTsNs over the shared response interface, replacing
the metadata-event-else-log-ts snippet copy-pasted at four ack sites.
- Extract acquireRenamePathLocks, replacing the verbatim sorted
two-path lock fence in both rename handlers.
* mount: no resurrection on foreign delete, version no-event acks, gate downloads, tighten copy adopt
Follow-ups to the review patches:
- Preserving dirty pages on a foreign delete let the next flush pass the
isDeleted guard and CreateEntry, resurrecting the remotely-unlinked
name. Mark the handle deleted in the vacate branch: the open fd can
still read its buffered writes, but a flush no longer recreates the
file.
- A no-event acknowledgment (log fence only) synthesized a metadata
event with TsNs 0, so the cache stored the entry unversioned and an
older subscriber event rolled it back. Stamp the synthesized event
with the ack's log position at all four ack sites.
- downloadRemoteEntry serialized its install but did not check the
version, so an older response arriving last overwrote the entry/base
while the monotonic version kept the newer value, fencing corrections
out. Install only when the response is at least as new as the handle.
- sameEntryContent compared only size and chunks, so a foreign chmod
with unchanged content was adopted as the copy's own event. Compare
everything except server-assigned timestamps, so a metadata-only
foreign change installs instead.
* mount: trim comments to the non-obvious why
The versioning work accumulated multi-line comment blocks restating what
the code says. Keep the constraint a reader cannot derive — why a fence
is exact, why a version must not advance without its value, why an
uncached parent's records fence nothing — and drop the rest.
* mount: distinguish rename from delete, tighten the download and adopt gates
- A rename emits a nil old-path invalidation just like an unlink, so the
vacate branch marked the handle deleted and later writes through the
already-open descriptor were skipped instead of persisted. Carry the
delete/rename distinction on the invalidation and mark only an actual
delete.
- The remote-download install accepted an unversioned response
regardless of the handle's version, so during a rolling upgrade a
delayed response could install stale content under a newer version.
Require the response to be at least as new, with one exception: a
handle still lacking local chunks takes the content anyway — it cannot
read without it — but does not claim the response's log position.
- Copy-event adoption returned without installing, so a foreign touch
arriving before the copy's own event lost its timestamps. Content is
unchanged either way, so the dirty pages stay valid; a clean handle now
takes the entry, while a dirty one keeps its diverged version.
* mount: one directory floor instead of a record per child; agree on TTL
Review feedback:
- Build completion wrote one KV record per direct child inside the cache
write lock, so a large directory stalled every other cache operation
for O(children) store writes. The directory's listing snapshot already
covers every child it saw; make that floor the version for any child
without a record of its own, and a child earns a record only when a
later event touches it. One map write per build replaces the per-child
writes, with the same fencing.
- The presence probe read the store directly and so counted a
TTL-expired entry as present, judging the path by a record describing
content that has logically vanished. It now applies the same expiry
the read path does, and an expired path falls back to its directory
floor.
- Preserve ErrNotFound identity when the commit-time re-read finds the
object deleted, so callers still surface a 404.
- Assert the rename-away source fence timestamp in the invalidation test.
Also record the tombstone ceiling: distinct deleted names in a cached
directory accumulate until it is rebuilt or evicted, which prunes
everything at or below the new snapshot.
* mount: pin the fence's clock domain instead of letting skew decide
A log-position fence is stamped by one filer's clock under that filer's
in-process lock, so comparing it to an event another filer logged is
comparing two unrelated clocks. The two error directions are not equally
costly: applying an event the fence already covered is a re-apply the
base-equality check absorbs, while skipping one it does not cover leaves
the handle holding exactly the state the event was meant to correct,
with the subscription cursor already past it — the unhealable staleness
this whole PR exists to remove.
So refuse to guess. Fences now carry the signature of the filer that
stamped them, and a handle records it alongside the position. An event
is only fenced out when the filer that logged it is the one that stamped
the fence — the logging filer appends its own signature, so its presence
identifies the clock domain. Events from any other filer are applied.
Positions taken from events keep comparing as before; the subscription
already delivers those in order.
The invalidation callback takes a struct now: it carries the path,
entry, position, delete/rename distinction, and signatures, and was
about to need a fifth positional parameter.
* mount: follow a foreign rename; key page invalidation on content, not equality
- A rename's old-path invalidation now carries the destination, and the
handle follows the file there: an open fd tracks the inode, and leaving
it on the old path made its next flush recreate that name instead of
updating the renamed file.
- Dirty pages overlay content, so only a content change invalidates them.
Keying that on exact equality meant any timestamp-only event destroyed
them, which the copy-adoption marker existed to paper over — a foreign
touch could consume the marker and leave the copy's own event to drop
the post-copy writes. Comparing content instead makes the marker
unnecessary, so it is gone: a metadata-only event keeps the overlay,
and a dirty handle keeps its diverged entry unless foreign content
supersedes it.
- A remote download response that is merely older is now refused even
when the handle still lacks chunks; only an unversioned one is taken
(and claims no position), since an older response's content predates
what the handle reflects.
- A refused or unversioned download no longer publishes to the metadata
cache, where a zero-position event would clear the entry's version and
let an older subscriber event roll the cache back.
* mount: page invalidation keys on content alone; unversioned writes claim no position
- sameEntryContent compared everything but timestamps, so a foreign
chmod, chown, or xattr change counted as a content change and
destroyed the dirty-page overlay. It was strict only to serve the
copy-adoption marker, which is gone; its one caller now asks the
question it actually needs — did the bytes change — so metadata-only
events leave the overlay alone.
- A rename over an existing file destroys that file, but its open handle
was left live and still pointed at the name the renamed source now
occupies, so its flush could overwrite it. MovePath already reports the
displaced inode; mark that handle deleted.
- An acknowledgment was refused whenever its position was numerically
lower, even when a different filer stamped the fence it lost to. Two
known, differing signatures mean unrelated clocks, so the comparison no
longer applies there; unknown signatures still compare as before.
- A local write with no log position behind it now records that
explicitly instead of deleting its version record. Absence means the
directory listing covers the path, which is why the snapshot floor
applies; local content the listing never saw must not inherit it, or
the events that would correct it are fenced out.
* mount: widen the existing lookup functions instead of forking WithVersion twins
The versioning work grew a parallel function for every accessor that
needed to return a log position — lookupEntryWithVersion beside
lookupEntry, maybeLoadEntryWithVersion beside maybeLoadEntry,
FindEntryWithVersion beside FindEntry, AcquireFileHandleWithVersion
beside AcquireFileHandle, advanceEntryVersion beside
advanceEntryVersionTsNs, plus a getPbEntryWithVersion wrapper and an
InsertListedEntriesForTest hook. Two names for one operation is two
places to keep in step, and the split let callers pick the one that
happened to compile.
Each pair is now the single original name carrying the position, with
callers that do not want it discarding it. filer_pb.GetEntry returns the
fence its response already carried rather than a mount-side wrapper
re-issuing the lookup, and InsertEntry takes the position its content
reflects rather than a test-only twin that inserted without one.
The one behavioural knot the merge exposed: AcquireFileHandle had been
installing the entry on a pre-existing handle only in its unversioned
form, which conflated 'the caller is authoritative' with 'the lookup had
no version'. Deferred create is the only caller that means the former,
so it now installs explicitly and the map function just acquires.
On renewal failure the renew goroutine stored isLocked=false before its
deferred renewGoroutineRunning.Store(false) ran. A concurrent RequestLock
interleaving there reacquires the lease (sees isLocked=false), sets
isLocked=true, then its CompareAndSwap on renewGoroutineRunning fails because
the old goroutine's flag is still set — so no replacement renewer starts. The
lock is then held locally with nothing renewing it, and silently expires on
the master after the lease TTL, admitting a second holder.
Clear renewGoroutineRunning before isLocked on the failure path so the
reacquire path always starts a fresh renewer. Builds and vets clean; no
behavior change on the success path.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
The copying and tagging tests force-drop each bucket's collection at the
master so volume slots are freed deterministically between tests. But the
copy-tests CI job runs its master on 9336 and the tagging Makefile on 9338,
while the tests default to 9333 — the cleanup dialed a dead port and quietly
no-oped. Each test bucket then grows 7 volumes against -volume.max=100, and
whenever async deletion lagged the data node ran out of slots and PutObject
500ed with "No writable volumes and no free volumes left".
Set MASTER_ENDPOINT where the master port is non-default: the copy-tests
workflow step, and the copying/tagging Makefiles (derived from MASTER_PORT).
* telemetry: validate reports on the collect endpoint
/api/collect is anonymous, so reports can't be authenticated, but a
real master can't produce a non-UUID topology_id, a version outside
N.NN(-enterprise), an unknown GOOS/GOARCH, or absurd counts — reject
those to keep casual junk out of the collected data, and cap the
request body at 4 KB.
* telemetry: integration test fixtures pass collect validation
The test's topology id and version were exactly the junk shapes the
new validation rejects; use a UUID and a plain version number.
telemetry: confirmed-cluster stats
Count a cluster as confirmed once it has reported on >=2 distinct UTC
days (per-cluster history makes this a length check). Version/OS
distributions in /api/stats are computed over confirmed clusters, so a
one-shot injected report can't appear in them; falls back to all active
clusters while no confirmed ones exist (fresh server). Adds the
seaweedfs_telemetry_confirmed_clusters gauge and a dashboard card.
telemetry: per-cluster usage history
Keep one compact sample per cluster per UTC day (disk bytes, volume
count, volume servers), retained for -max-age and persisted in the
state file. Serve it at /api/history?cluster_id=...&days=90 and add a
per-cluster lookup with disk/volume charts to the built-in dashboard.
* telemetry: persist server state across restarts
The telemetry server kept the instance map and Prometheus gauges only
in process memory, so every deploy or restart reset all collected
metrics until clusters re-reported over the next 24h.
Snapshot the instance map to a JSON state file (atomic tmp+rename) on
a debounced interval and on SIGTERM, and restore it on startup,
preserving received_at so the cleanup and active-cluster windows stay
correct. Defaults to data/telemetry-state.json, which the deployed
systemd unit's WorkingDirectory already provides; -state-file=''
disables.
* telemetry: keep instances for 90 days by default
With state now persisted across restarts, a longer retention default is
meaningful; raise -max-age from 30 to 90 days so per-cluster data
survives long enough for quarterly views.
ci: build telemetry server from its own module in deploy workflow
telemetry/server has had its own go.mod since #9924, so building
./telemetry/server/main.go from the repo root fails with 'no required
module provides package'. Build from within the module instead.
telemetry: key value gauges by cluster_id only
The value gauges were labeled {cluster_id, version, os}, so a cluster
reporting back after an upgrade started a new series while the old one
kept its last value forever: sum() double-counted every upgraded
cluster, and per-cluster history broke at every version change.
Key the five value gauges by cluster_id alone so each cluster keeps one
continuous series across upgrades; version/os metadata stays on
cluster_info (deleted and re-set on change), available to value queries
via 'on(cluster_id) group_left' joins. Update README accordingly.
telemetry: aggregate /api/metrics per day so dashboard charts render
The dashboard expects {dates, server_counts, disk_usage} as parallel
arrays, but GetMetrics returned per-instance {date,value} lists under
different keys, so the two over-time charts never rendered.
* Add GitHub Actions workflow for codespell on master
* Add rudimentary codespell config
* Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms
Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers
like allLocations, publishErr, ReadInside, FlushInterval. Also skip
templ-generated *_templ.go files, and whitelist a handful of
short/domain-specific words (visibles, fo, te, ser, bject, unparseable,
keep-alives, tread, anc, ue) that show up as false positives across the
tree.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix ambiguous typos and protect false positives
Fixes typos that codespell reports with multiple candidate suggestions
(so `codespell -w` cannot auto-apply them), plus one inline pragma and
one config entry to protect legitimate identifiers.
Manual fixes (single correct answer chosen from context):
- pattens -> patterns (5x) in filer/upload/shell flag help strings
- finded -> found (2x) in tarantool storage.lua comment
- spacify -> specify (2x) in helm chart values.yaml comment
- wether -> whether in skiplist.go docstring
- simpe -> simple in mq schema test case name
False-positive protection:
- Add `//codespell:ignore` next to `source GET's` (possessive of HTTP
verb) in s3api_object_handlers_copy_stream.go
- Whitelist `auther` in .codespellrc — it's a local variable meaning
"authenticator" in weed/security/tls.go, not a typo of "author".
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Extend codespell ignore list: .git-meta path and thirdparty groupId
Also skip `.git-meta` (scratch dir for commit messages that may contain
typo words verbatim) and whitelist `thirdparty` — it appears as the
literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms
and cannot be renamed.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w
Auto-applied fixes to the 44 remaining single-suggestion typos across
docs, comments, log messages, tests, config, and one Java pom.
=== Do not change lines below ===
{
"chain": [],
"cmd": "uvx codespell -w",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
* Revert breaking codespell fixes; whitelist unknwon and atleast
Two of the auto-applied `codespell -w` fixes were false positives that
would break the build/tests:
- go.mod: `github.com/unknwon/goconfig` is a real Go module path — the
upstream author's GitHub handle is literally `unknwon`. Renaming to
`unknown` would fail dependency resolution.
- test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}:
`atleast` is a literal CLI mode value (a string constant compared and
passed as a positional argument). Rewriting to `at least` splits it
into two arguments and breaks the mode check.
Reverted those files and whitelisted both words in .codespellrc so
future runs won't re-suggest the same broken fixes.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(admin): implement ApplyPluginConfigFromToml to propagate settings to plugin config store
* Update weed/admin/dash/config_toml.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* admin: overlay admin.toml onto plugin configs through bootstrap defaults
Creating a config from scratch at startup skipped the descriptor-defaults
bootstrap, so a job type with only worker keys in admin.toml persisted
Enabled=false and RetryLimit=0 and silently stopped running. Overlay
existing configs at startup, and apply the same overlay in
enrichConfigDefaults when the plugin bootstraps a fresh config from
descriptor defaults.
Also place collection_filter in the admin values where workers read it,
map preferred_tags as a string list, and stamp UpdatedAt.
* admin: trim the admin.toml help text and call-site comment
* admin: clamp toml retry values to the int32 range
* admin: fail startup when admin.toml cannot reach the plugin config
The legacy overlay already aborts startup when declared settings cannot
persist; continuing here would let workers bootstrap with stale values.
* admin: fix the retry clamp test on 32-bit
A 32-bit int cannot hold the oversized toml value, so viper returns 0
before the clamp runs.
---------
Co-authored-by: baracudaz <baracudaz@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume.fix.replication: add a well-placed replica before deleting a misplaced one
A misplaced volume with no surplus replica (replica count == copy count) used to have its misplaced replica deleted first, and the replacement copied only on the next pass. That drops the volume below its intended durability, and permanently so when no destination can accept the replacement copy. Now such volumes get a well-placed replica first, and the misplaced one is trimmed as surplus on a later pass, following the same fulfill-before-delete principle as volume.tier.move (#8950).
The classification switch is extracted into classifyReplicaSet so the add-wins-over-trim priority is unit-testable.
Also stop the fix loop when an -apply pass makes no progress (nothing copied, nothing deleted); previously an unplaceable under-replicated volume made the loop spin forever, re-collecting the topology every 15 seconds. And propagate ewg.Wait() errors instead of swallowing them.
* volume.fix.replication: drop unused allLocations from deleteOneVolume
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: generate the SFTP host key per install
The SFTP secret template shipped one fixed ed25519 host key, so every
install that did not override it presented the same host identity.
Generate the key at install time instead, following the
getOrGeneratePassword pattern: an existing secret keeps its key across
upgrades, except the previously bundled one, which is replaced with a
freshly generated key on the next upgrade.
* helm: create the SFTP host-keys secret the deployments mount
Both the sftp and all-in-one deployments mount /etc/sw/ssh from
<fullname>-sftp-ssh-secret, but no template created it, so a default
install could not start its pod and host keys only reached the server
when enableAuth happened to mount them elsewhere. Create the secret
with a generated ed25519 key, keeping whatever keys an existing secret
already holds. The sshPrivateKey default becomes empty: the file it
pointed at only exists when enableAuth mounts /etc/sw, and a configured
but missing key file is fatal to the server, while hostKeysFolder now
always has a key.
* helm: test SFTP host key generation and secret lifecycle
Template checks: keys render into the secret the deployments mount,
parse as PKCS#8 ed25519, differ between installs, and the render
carries no key material from the chart itself; existingSshConfigSecret
and all-in-one wiring covered. On the kind cluster, exercise the
secret lifecycle: a generated key survives upgrades, the key earlier
chart versions bundled is replaced, and operator-managed keys are kept
untouched. chart-testing now also installs with sftp enabled, where
the pod only becomes ready if the server loads the generated host
key.
* helm: treat a whitespace-only stored SFTP host key as missing
A whitespace-only secret value skipped regeneration and then rendered
an empty key file.
* helm: mount the SFTP host keys secret at the configured hostKeysFolder
The secret was mounted at a fixed /etc/sw/ssh, so a custom
sftp.hostKeysFolder pointed the server at an empty directory. Mount at
the configured path in both the sftp and all-in-one deployments, and
pin flag/mount agreement in the rendering tests.
* s3: track manifest blob ownership through multipart completion
Manifest blobs made three orphan paths. A partial fold that failed midway
kept its earlier batches on volume servers while the write fell back to
flat chunks; the fold now records each saved blob and deletes them on
error. A completion that failed after preparing left its fresh manifests
behind on every retry; the completion state now owns them and deletes
them unless a failed rollback left the version entry still holding them.
And a completed upload removes its parts metadata-only, which stranded
the part-manifest blobs superseded by flattening; those are collected
during flattening and deleted once the completion commits.
Two shared-chunk hazards nearby: the version-file rollback deleted its
data, destroying the still-registered parts (worse once manifests
resolve to inner chunks), and the idempotent-replay cleanup data-deleted
leftover parts whose chunks the live object references. Both are
metadata-only now.
* s3: trim chunk manifest comments
* s3: test manifest fold rollback and part range selection
The fold-with-rollback and the boundary-to-byte-range logic were only
exercised by hand against a live server; give both an injectable seam
and cover the fold, the below-threshold and SSE no-ops, the midway
failure deleting the blobs it saved, and offset-vs-legacy-index range
selection including indexes that no longer address the chunk list.
* erasure_coding: WriteDatFile takes the encode-time dat size for the shard block layout
* volume server: derive EC decode layout from the encode-time dat size, not the live extent
* erasure_coding: test decode after tail deletions shrink the live extent below a large-block row
* seaweed-volume: write_dat_file_from_shards takes the encode-time dat size for the shard block layout
* seaweed-volume: derive EC decode layout from the encode-time dat size, not the live extent
* seaweed-volume: test decode after tail deletions shrink the live extent below a large-block row
* erasure_coding: reject decoding with no data shards
* worker: record the encode-time dat size in the .vif
* erasure_coding: fall back to the shard-derived layout only when the encode-time dat size is missing
* erasure_coding: reject an ambiguous shard-derived block layout
* seaweed-volume: fall back to the shard-derived layout only when the encode-time dat size is missing
* seaweed-volume: reject an ambiguous shard-derived block layout
* filer: accept a WriteCondition on UpdateEntry, under the per-path lock
UpdateEntry was a bare read-modify-write: the precondition check, the
chunk garbage diff, and the store write could interleave with a
concurrent update to the same path. Take the per-path lock CreateEntry
already holds, and evaluate an optional CreateEntry-style WriteCondition
under it, failing with FailedPrecondition like expected_extended.
* filer: IF_CHUNKS_EQUAL write condition compares the stored chunk fid set
A chunk-preserving read-modify-write (tagging, setattr, copy-in-place)
races UpdateEntry's garbage diff: if a concurrent update empties the
chunk list first, the stale writer's commit resurrects fids that are
already queued for deletion, stranding the entry on a dead needle once
vacuum reclaims it. The reverse also holds: a writer that read an empty
chunk list can wipe chunks a concurrent update just added.
IF_CHUNKS_EQUAL guards both: the stored chunk fid multiset must still
equal what the caller read, order-independent, with an empty fids list
expecting no chunks. Absent entry counts as no chunks for CreateEntry
overwrites and transactions.
* filer: delete and append serialize on the entry path lock
DeleteEntry queues the entry's chunks for deletion and AppendToEntry
rewrites the chunk list, but neither held the per-path lock, so either
could interleave with a conditional update between its precondition
check and its write — a passed IF_CHUNKS_EQUAL would then resurrect
fids already on the deletion queue, or clobber a freshly appended
chunk. AppendToEntry keeps the cluster lock for cross-filer append
serialization; the path lock covers the local read-modify-write.
* filer: reuse lockPath in UpdateEntry lookup
s3: fold large chunk lists into manifest chunks on the direct write path
The S3 gateway uploads chunks itself and hands the filer a fully prepared
entry. On the routed write path (ObjectTransaction) the filer stores that
entry as-is, so a large PutObject or CompleteMultipartUpload persisted its
whole flat chunk list - a 900GB object carries 120k chunk references in one
entry. Manifestize on the gateway before the entry is written, the same way
mount, WebDAV, and filer.copy prepare theirs.
Multipart part boundaries also record byte offsets now: the stored chunk
indexes stop matching the entry once the list is folded, and partNumber
reads plus GetObjectAttributes prefer the offsets. Legacy index-only
records still work, with bounds checks instead of a possible panic.
Copy paths resolve a manifested source into data chunks before their
per-chunk copy loops - copying a manifest chunk raw would store its blob
as object data still pointing at the source - and a large copied list is
folded again on the destination. Completion likewise resolves manifest
chunks a part entry may carry (the filer folds an oversized UploadPartCopy
range) before rebasing part offsets.
The maintenance operations built manifest, manifest-list, data-file and
metadata-log paths with a bare path.Join("metadata", ...), so a single
maintenance run wrote scheme-less relative paths into the new snapshot,
the metadata-log and the table xattr. SeaweedFS reads those back fine
because normalizeIcebergPath accepts both forms, but strict readers
resolve every location through S3FileIO and fail with "Invalid S3 URI,
cannot determine scheme", leaving the whole table unreadable after any
maintenance commit.
Add absoluteIcebergPath, the inverse of normalizeIcebergPath, and apply
it at every site that authors locations: rewrite_manifests, compact,
rewrite_position_delete_files, and the shared commit path. The base is
derived from the bucket and table path since that is where the worker
physically writes, matching the locations the REST catalog generates.
The pre-move stale check compared the master's location urls (host:port)
literally against the proposal's node addresses, which carry the grpc
suffix (host:port.grpcPort). Every move failed as "stale move: volume no
longer on source" even though the volume was still there. Normalize both
sides through pb.ServerAddress before comparing.
* regenerate master_grpc.pb.go with protoc-gen-go-grpc v1.6.2
The other generated pb files are already on v1.6.2; this one was stale.
* shell: keep unlock from racing the lease renewal
A renewal RPC in flight while ReleaseLock runs re-creates the lock on the
master after the release deletes it, and can blank the client name if the
renewal reads it mid-release. The stale-token release is then ignored, so
the lock stays held (sometimes anonymously) until it expires. Serialize
the renew and release RPCs, and set the client name before flipping
isLocked so the renewal never sends a partial acquisition.
* shell: restart lease renewal after a failed renewal
The renewal goroutine exits on error but never cleared its running flag,
so later locks in the same process were never renewed and silently
expired after ten seconds.
* shell: show who holds the cluster lock
A blocked lock command gave no hint that another client holds the lock
(the refusals only surfaced at -v=2), and cluster.status reported the
shell's own lock state as if it were the cluster's. Add a
GetAdminLockStatus RPC to the master so lock prints the holder before
blocking and cluster.status shows the actual cluster-wide holder. Both
degrade silently against masters without the RPC.
* shell: bound admin lock RPC attempts with timeouts
The lease, renew, release, and holder-status calls all ran without a
deadline, so an unresponsive master could hang the renewal goroutine,
an unlock (which now waits on the renewal mutex), or the shell prompt.
Give each attempt its own short context; the retry loops still resolve
a fresh leader on the next try.
* master: reject admin token release on non-leaders
A follower holds no lock state, so it answered a release with success
while the leader kept the lock until expiry. Refuse like LeaseAdminToken
does so the client can try the leader instead.
* shell: leave the lock release call unbounded
A release cut short by a deadline leaves the lock held on the master
until it expires, so a slow master would turn every unlock into a
ten-second ghost lock. Restore the single fire-and-forget attempt;
the timeouts stay on the lease and renew paths, where a stalled call
forfeits the lease anyway.
* shell: release only the token unlock started with
A RequestLock racing a slow release (the admin presence lock does this
on shutdown) could have its freshly acquired token sent in the release
request or zeroed by the trailing stores. Capture the token once under
the mutex and compare on clear so a concurrent acquisition survives an
in-flight unlock.
Reads of remote-backed entries now record hit or miss in
SeaweedFS_remote_cache_read_total{source,bucket,result} on the filer HTTP
path and the S3 gateway, so cache effectiveness of mounted buckets can be
graphed. Inline-content entries count as hits since they are served
locally without chunks. The filer purges the per-bucket series when the
bucket directory is deleted, so a standalone filer does not accumulate
series across bucket delete/recreate churn.
Only the heartbeat path read the .dat mtime; collectStatForOneVolume left
the field at 0, so /status consumers could not tell how long a volume had
been idle.
Commit 8bff3b32 changed BatchDelete to keep processing after a cookie
mismatch but left the integration test asserting the old early-break
behavior, breaking Volume Server Integration Tests (grpc - Shard 1) on
master. Align the test with the new semantics and port the same
break->continue to the Rust volume server, which runs the same suite
via VOLUME_SERVER_IMPL=rust.
The initiator's shed check (initiatedGrow != HasGrowRequest) compares
against an err from a PickForWrite that may predate the growth
concluding: the grower registers its volumes before clearing the flag,
so when growth lands between the failed pick and the check, the assign
shed ResourceExhausted even though a writable volume was already
registered. Re-pick once after observing the conclusion and shed only
if the volume layout still has nothing writable. Applies to both the
gRPC Assign and the HTTP dirAssign paths, which share the shed logic.
Flaked in CI as TestAssignInitiatorWaitsForItsOwnGrowth; reproduced
deterministically by widening the enqueue-to-check window.
* fix: report short S3 ReaderAt reads
Problem: S3BackendStorageFile.ReadAt returned a short buffer with a nil error, hiding truncated remote data.
Root cause: Every terminal io.EOF was cleared regardless of how many bytes were read.
Fix: Clear io.EOF only when the requested buffer was completely filled.
Validation: go test ./weed/storage/backend/...
Co-authored-by: Codex <noreply@openai.com>
* fix: validate S3 ReaderAt requests
Co-authored-by: Codex <noreply@openai.com>
* s3 ReadAt: simplify negative offset error
* s3 ReadAt: stop reading once the buffer is full
Skips the extra Read that only collected the terminal EOF, and bounds
the loop if the server ignores the Range header and returns more data
than requested, where Read on an empty slice can spin forever.
* s3 ReadAt: cover full reads that arrive with io.EOF
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix: reject overflowing needle ID deltas
Problem: Parsing a file ID with a delta can wrap a valid maximum needle ID back to zero without returning an error.
Root cause: Needle.ParsePath added the parsed uint64 delta without checking whether the sum exceeded the needle ID range.
Fix: Compare the delta with the remaining uint64 capacity before addition and return a contextual overflow error when it does not fit.
Validation: go test ./weed/storage/needle -run ^TestNeedleParsePathRejectsDeltaOverflow$ -count=1; go test ./weed/storage/needle -count=1; git diff --check 10cdaf381875492a2c752d1038797e96ff18208f..HEAD
Co-authored-by: Codex <noreply@openai.com>
* fix: propagate needle ID delta parse errors
Co-authored-by: Codex <noreply@openai.com>
* print the needle id in hex in the delta overflow error
* batch delete: keep processing after a cookie mismatch
* rust volume: reject overflowing needle id deltas
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The dev container workflow prebuilds the Rust volume server from
seaweed-volume/ but did not watch that directory, so Rust build fixes
landed without this workflow validating them.
pass --experimental_allow_proto3_optional to protoc in the Rust build
filer.proto now carries a proto3 optional field, which the protoc 3.12
shipped in ubuntu-22.04 apt rejects unless this flag is set. Newer
protoc versions still accept the flag, so it is safe everywhere.
When every volume is full, a chunk upload fails only after the assign
retry budget, the failed chunk's data is dropped, and the mount kept
accepting writes anyway. cp would crawl for hours pushing the rest of
the file through a pipeline that could not persist it, and only close()
reported an error - a generic EIO.
Poison the file handle on the first failed chunk upload so subsequent
writes fail immediately, and map "no writable volumes" / "no free
volumes" upload errors to ENOSPC so the writing process aborts with
"No space left on device". Also guard lastErr with a mutex: it was
written concurrently by uploader goroutines, and keep the first error
so later failures do not mask the root cause.
* remote storage: carry Content-Encoding into mounted entries
A RemoteEntry now records the remote object's Content-Encoding, and every
path that materializes a local entry from remote metadata (lazy fetch, lazy
listing, remote.mount, remote.meta.sync, remote.cache) stamps it into the
entry extended attributes, so HTTP and S3 HeadObject/GetObject return the
header. GCS and Azure populate it on listing and stat; S3 only exposes it
via HeadObject, so listings leave it empty.
* remote storage: set Content-Encoding when uploading to the remote
An entry carrying Content-Encoding in its extended attributes (a native S3
upload, or a value pulled from the remote) now keeps it when
filer.remote.sync or remote.copy.local writes the object to GCS, S3, or
Azure, instead of silently dropping it.
* gcs: read remote objects without decompressive transcoding
GCS transparently decompresses gzip-encoded objects on download, which
ignores range requests and returns byte counts that disagree with the
tracked RemoteSize. Request the stored bytes instead; chunked reads of
gzip-encoded objects then behave like any other object.
* remote storage: track Content-Encoding presence so removals propagate
A listing that does not report encodings (S3) leaves the field unset and
the local header untouched, while an authoritative report of no encoding
(GCS, Azure, any stat) now clears a previously stamped header instead of
leaving it stale. remote.cache also schedules a metadata update when only
the reported encoding changes.
* remote storage: propagate Content-Encoding on metadata-only updates
filer.remote.sync routes same-content changes through UpdateFileMetadata,
which only touched custom metadata (GCS, Azure) or tags (S3), so a
Content-Encoding change in the extended attributes never reached the
remote object's real header. GCS now patches contentEncoding alongside
the metadata, and Azure reissues the blob's HTTP headers with the new
value, carrying the others over since the call replaces the full set.
S3 stays tags-only: changing the header there means rewriting the
object, which the sync already does whenever content changes.
* remote.meta.sync: optional per-file stat for listing-omitted metadata
S3 listings carry no Content-Encoding, so entries synced from them never
learn it and the lazy-stat path never runs once an entry exists. With
-statFiles, each new or changed file whose listing left the encoding
unreported is stat-ed before reconciling, and the stat-derived value is
persisted so the next run only stats files that changed. Off by default:
it costs one remote request per file, and GCS and Azure listings already
carry the encoding.
* s3: apply metadata-only Content-Encoding changes with an in-place copy
Content-Encoding is S3 system metadata, so the tags-only metadata update
silently left the object's real header untouched. When the encoding
differs, reissue the object as a self-copy with replaced metadata,
carrying the content type and configured storage class like a fresh
write does. CopyObject caps at 5 GiB; beyond that the change is logged
and applies on the next content write.
* azure: skip the metadata call when user metadata is unchanged
An encoding-only change reissues the blob's HTTP headers; sending the
unchanged user metadata alongside it wastes a round trip and bumps the
blob's ETag once more than needed.
* s3: carry existing object metadata through the encoding copy
The replace directive drops everything not resent, and a mounted entry
usually has no local mime or user metadata, so the in-place copy wiped
the object's Content-Type, Cache-Control, user metadata, encryption
settings, and storage class. Read them back with a HeadObject first and
carry them over, overriding only what SeaweedFS manages: the encoding,
a locally set mime, and the configured storage class. S3 reports
Expires as a string while the copy input wants a time, so it is parsed
and skipped when malformed.
* generate vtproto marshalers for filer_pb and use them on the metadata log path
Reflection-based proto.Unmarshal allocates a fresh message tree through
reflect.New on every call. On the metadata subscription fan-out the same
event is decoded once per subscriber, so reflect.New tops the decode
churn under many mounts.
Generate MarshalVT/UnmarshalVT/SizeVT for filer.proto (a separate
filer_vtproto.pb.go, filer.pb.go untouched) and call them on the log
entry marshal and the subscribe/replay decode paths. UnmarshalVT
allocates message structs directly and copies byte and string fields, so
it stays wire-compatible with proto.Unmarshal and preserves the
non-aliasing the persisted-log cache depends on.
For SubscribeMetadataResponse this cuts decode allocations 69 -> 50 and
~4.5us -> ~2.1us per event; the win scales with subscriber overlap.
* marshal log entries directly into the buffer
SizeVT is allocation-free and MarshalToSizedBufferVT writes into a
pre-sized slice, so the log entry can be marshaled straight into
logBuffer.buf. This drops the per-entry MarshalVT allocation and the
follow-up copy on the write path.
* expand vtproto benchmarks: marshal, decode, and marshal-into-buffer by chunk count
Parametrize by nested-message count (chunks per event) and add encode +
zero-alloc marshal-into-buffer benchmarks alongside the decode one, so
the write-path win from MarshalToSizedBufferVT is measurable too.
* keep proto.Unmarshal for metadata events to preserve UTF-8 validation
UnmarshalVT skips proto3's UTF-8 validation of string fields, so a
SubscribeMetadataResponse with an invalid-UTF-8 string (e.g. Directory
"\xff") that proto.Unmarshal rejects would decode and reach path
filtering and subscribers. Decode events with proto.Unmarshal again;
UnmarshalVT stays on the log entry paths, whose only variable-length
fields are bytes and so carry no UTF-8 constraint.
Tests cover the codec difference and that a malformed event is skipped
before delivery.
Put/DeleteObjectTagging set the update directory to the bucket root for a
null version, so a tag change on allowed/protected.txt landed on
protected.txt at the bucket root instead. A principal scoped to one
nested key could overwrite a different object sharing the basename, the
same class of scope bypass fixed for PutObjectAcl. Share the object's
parent-directory resolver across both paths.
* route log buffer flush copies through the shared slab pool
Each flush copied the sealed window into a bytes.Buffer drawn from a
package-local sync.Pool. GC drains that pool, so once collections run
often (e.g. under GOMEMLIMIT) most flushes miss and Write grows a fresh
window-sized array, the dominant bytes.growSlice source under write load.
Copy into a size-classed slab from weed/util/mem instead. Slabs are
reused process-wide and returned to their exact size class after the
flush, so variable-sized flushes across many partitions stop churning
mismatched buffers.
* guard nil slab in flush releaseMemory
mem.Free(nil) resolves to the smallest slot pool and stores a zero-cap
slice, so a later mem.Allocate could return nil and panic. The current
call sites never pass nil, but the guard keeps a double release harmless.
The FUSE harness picks the filer HTTP port and lets "weed mount" derive the
filer gRPC port as HTTP+10000. freePort kept the HTTP port under the Linux
ephemeral floor (32768) but not the derived gRPC port, which ranged up to
42000. When the gRPC port landed above the floor an outbound connection could
transiently hold it, so mini relocated its filer gRPC port while mount kept
dialing HTTP+10000, timing the mount out.
Cap the HTTP port at 22000 so the gRPC sibling stays at or below 32000, and
verify both ports are bindable before returning.
* Fix PutObjectAcl writing back to the wrong object for nested keys
PutObjectAcl set the update directory to the bucket root, so an ACL
change on allowed/protected.txt landed on protected.txt at the bucket
root instead. A principal scoped to one nested key could overwrite a
different object sharing the basename. Target the object's own parent
directory.
* Test object ACL update directory resolution for nested keys
* Introduce weed shell command `ec.check.replication`.
This command performs a quick check of EC volume shard replication, reporting
volumes whose shards are over- or under-replicated. Each volume is checked
against its own data+parity ratio, obtained via
erasure_coding.EcShardsVolume{Data,Parity}Shards, so builds that derive the EC
ratio per volume report custom ratios correctly.
The name follows the shell's convention (cluster.check, volume.check.disk); the
closest normal-volume counterpart is volume.fix.replication.
* shell: ec.check.replication reports mixed under+over-replication in both lists
* rust volume: verify the .dat ends at the last indexed needle
The Go loader quarantines a volume whose .dat extends past the last
indexed needle - the leftover of a torn shutdown - but the Rust check
only verified header fields of the trailing index entries and never
compared file sizes, so a torn tail loaded clean and writable. Appends
land at the raw file end, and past a misaligned tail the next needle
sits at an offset the 8-byte-unit .idx encoding rounds down, pointing
the index a few bytes before the needle.
Replace the last-10-entries walk with the current Go shape: find the
entry physically last in the .dat (append-ordered fast path, max-offset
scan for key-sorted rebuilds), verify that needle - tombstones with
their on-disk Size=0 - and require the file to end exactly at it,
marking the volume read-only otherwise.
Claude-Session: https://claude.ai/code/session_01XgGXMLknzaNgQzHMyo2Vhb
* rust volume: buffer the .idx max-offset scan
The slow path read one 16-byte entry per syscall; a BufReader batches
the sequential scan like Go's WalkIndexFile does.
Claude-Session: https://claude.ai/code/session_01XgGXMLknzaNgQzHMyo2Vhb
StatFs on a mount reported cluster-wide total capacity but used size from
a single volume layout keyed by collection, replication, ttl, and disk
type. A mount without -collection therefore showed only the default
collection's usage, hiding data in named collections, and even a
collection-scoped mount missed volumes with a different replication,
ttl, or disk type.
Aggregate used size and file count across all layouts of the requested
collection, and across every collection when the collection is empty,
matching how Topology.Lookup treats an empty collection. Looking up
stats no longer creates a phantom collection as a side effect.
An absent needle is normally indistinguishable from a vacuumed delete,
so check.disk skips it and replicas diverged by replication failures
cannot be reunited. But a replica with compaction revision 0 has never
been vacuumed: every delete it processed still holds its tombstone, so
a needle absent there is provably a missing write.
The new flag resurrects absent needles in exactly that case. The
receiving replica's live compaction revision is read after the index
snapshot and must be 0; vacuumed replicas keep the skip, now with the
revision in the message. The decision is passed per direction since
pass 2 runs pairs concurrently. Resurrected needles count toward
-nonRepairThreshold as before.
* filer: authorize TUS existing-session verbs against the validated stored target
Scope-check on TUS HEAD/PATCH/DELETE only populated a resource path for POST,
so a prefix-restricted token that learned another tenant's session id could act
on that session and land content at a TargetPath its own AllowedPrefixes forbid.
Split the filer JWT check into authenticateFilerJwt (signature and method) and
authorizeFilerJwtPaths (resource scope), and make the scope check fail closed:
a prefix-restricted token with no resolved resource path is denied instead of
authorized on signature alone. The TUS handler now authenticates first, reads
and validates the session once, authorizes the stored TargetPath, then operates
on that single pinned snapshot. readTusSessionInfo rejects a session whose id,
target or size is unusable, and getTusSession is split so the authorization
lookup no longer lists chunks.
* filer: reject non-canonical TUS upload ids
The uploads route took the first path component as the session id, so a trailing
path or other non-canonical spelling aliased one session under several URLs.
Require the id to be a canonical UUID, the only form the server mints, both at
routing and when reading a session's metadata, so one URL maps to one resource.
* filer: revalidate the pinned TUS session before completing an upload
Completion re-read chunks but not the session identity, so a PATCH finishing
after a concurrent DELETE or metadata replacement could still land at the id's
stored path. Before completing, confirm the session still exists and its target,
size and creation time are unchanged from the authorized snapshot; otherwise the
completion fails instead of writing to a path the request never authorized.
* filer: log TUS session lookup failures before returning not-found
readTusSessionInfo and loadTusSessionChunks failures answered "not found" with
no log line, so a transient filer or listing error was indistinguishable from a
genuinely missing session. Log the lookup at V(1) (a missing session is common
and benign) and the chunk-load error at Errorf (the session already resolved).
* volume: copying a remote-backed volume only needs space for the index
VolumeCopy sized its target-location check by the source .dat even when
that .dat lives in a cloud tier and only .idx/.vif land locally, so
re-replicating a tiered volume demanded the full remote size in free
disk. Require the index size instead.
* shell: volume.tier.upload keeps volume replicas
Tiering a replicated volume deleted every replica but the upload
source, leaving one server holding the only .idx and the only .vif
that knows the remote object key — losing that server orphaned the
volume even though its data sat intact in the cloud.
Replicate the uploaded .idx/.vif onto the other replica servers
instead (VolumeCopy skips the .dat for remote-backed volumes), so all
replicas serve reads from the same remote object and the volume keeps
its replica count. An already-tiered replica is preferred as the
upload source, so a rerun after a partial failure reuses the existing
remote object instead of uploading a second copy under a new key.
* shell: group tier upload locations instead of re-prepending
* rust volume: copying a remote-backed volume only needs space for the index
Mirror the Go VolumeCopy change: size the free-location check by the
source .idx when the .dat lives in a cloud tier, since only .idx/.vif
land locally.
* admin: surface lifecycle rule counts in bucket listing
* admin: add bucket lifecycle JSON endpoint
* admin: show lifecycle rules on the buckets page
* admin: make the lifecycle count badge keyboard-accessible
* admin: match buckets empty-state colspan to the column count
* admin: drop stale lifecycle modal responses
* make
* sftp: match permission paths on path-component boundaries
Permission checks compared paths with raw string prefixes, so a
permission entry for /tenants/alice also matched sibling paths such as
/tenants/alice-archive, letting a scoped user read and overwrite
another tenant's files. The home directory containment check had the
same flaw. Route both through pathWithin, which cleans the paths and
requires exact equality or a separator-delimited descendant.
* sftp: clean permission path into a local when ranking matches
* filer: allow clearing a bucket read-only flag stuck after quota removal
Once s3.bucket.quota.enforce marks a bucket read-only, removing or
disabling the quota orphans the flag: enforcement skips buckets with a
non-positive quota, and fs.configure merges booleans with OR so
-readOnly=false could never turn it off. The only way out was deleting
the whole path rule.
- s3.bucket.quota -op=remove/disable and the admin server quota update
now lift the read-only flag on the bucket's path rule
- fs.configure now honors explicitly passed false boolean flags
(-readOnly, -fsync, -worm) instead of OR-merging them away
* filer: ClearBucketReadOnly reports unchanged when the save fails
* fix(rust-volume): remove .ecsum sidecars on EC destroy / shard delete
Rust EcVolume::destroy removed shards and .ecx/.ecj/.vif but left bitrot
checksum sidecars (.ecsum / .ecsum.v*). On clusters that run weed-volume
(not Go weed volume), collection.delete therefore orphans every sidecar
while correctly wiping shards — observed live on 4.39 (14/14 .ecsum
survived after collection.delete on a freshly encoded EC volume).
Go Destroy already calls RemoveBitrotSidecars; this brings Rust to parity:
- hoist remove_bitrot_sidecars into ec_bitrot (shared helper)
- call it from EcVolume::destroy for dir / dir_idx / ecx_actual_dir
- call it from Store::delete_ec_shards when a disk has no remaining shards
- unit test: test_destroy_removes_bitrot_sidecar
* rust volume: gate the shard-delete sidecar sweep on a local shard removal
Only sweep a disk's .ecsum when this delete actually removed a shard file
there, matching Go's found gate: a delete that never touched a disk must not
strip a sidecar it does not own — a shared -dir.idx sibling with surviving
shards, or an ec.rebuild index-prep copy that lands .ecx/.ecsum before any
shard. The shard-presence probe now treats unexpected stat errors as
"exists" so a transient failure cannot orphan-classify live shards, and
check_all_ec_shards_deleted reuses it.
* rust volume: destroy() sidecar sweep needs only the data and idx bases
ecx_actual_dir is always one of the two, so the third branch could never
run; this is now exactly Go Destroy()'s two-base sweep.
* rust volume: call the shared sidecar removal helper directly
* rust volume: unit-test remove_bitrot_sidecars
Mirrors Go's TestRemoveBitrotSidecars: legacy and versioned sidecars are
removed, a shard file and a longer-vid sidecar survive, absent is success.
* rust volume: keep the shared idx-base sidecar while a sibling disk has shards
One -dir.idx serves every location, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards. Nothing
reads the idx-base sidecar today, but .ecx shows index-dir files are real;
this keeps the defensive sweep safe if a writer ever lands one there.
* ec shard delete: keep the shared idx-base sidecar while a sibling disk has shards
One -dir.idx serves every disk, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards of the
volume — the same gate the Rust volume server applies. A status error counts
as in-use so a transient failure never strips it early.
* rust volume: drop a shard-only disk's stale .vif with the node's last shard
Go's removeEcSharedIndexFiles also clears the data-base .vif in the
all-shards-gone pass, gated on .idx absence so a disk still hosting the
source volume keeps its live .vif; the Rust delete path left it behind.
Unexpected stat errors count as .idx-present so a transient failure never
strips a live volume's .vif.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: scope TUS HEAD/PATCH/DELETE against the session target path
checkTusJwtAuthorization only populated the scoped-path list for POST, so
a prefix-restricted token was scope-checked at session creation but not on
HEAD/PATCH/DELETE, which act on an existing session addressed by id. A
low-privilege tenant holding a valid write token who learns another
upload's session id could PATCH attacker bytes into that session, or
DELETE/HEAD it, landing content at a target path its own AllowedPrefixes
forbids. The session id is unguessable to an unauthenticated attacker but
is not an authorization boundary against a legitimate tenant.
Resolve the effective target for HEAD/PATCH/DELETE from the session's
stored TargetPath and scope the prefix check against it, the same way POST
scopes the create target. The scoped paths are resolved lazily so the
session read runs only for a prefix-restricted token; unrestricted tokens
and deployments without a signing key touch no extra state. An unknown
session stays unscoped so the handler still answers 404.
* filer: fail closed when a TUS session target cannot be resolved
The lazy scope resolver swallowed every readTusSessionInfo error and left
the request unscoped, so a filer read error or a corrupt session .info
would authorize a prefix-restricted token against a target the server
never resolved. Only a genuinely missing session should stay unscoped (so
the handler answers 404); any other failure now propagates and denies the
request. checkJwtAuthorizationScoped's resolver returns an error and a
failed resolution is treated as not-authorized.
* filer: fold the scoped-path resolver into checkJwtAuthorization
checkJwtAuthorization now takes the lazy resolver directly instead of a
separate checkJwtAuthorizationScoped wrapper, and the surrounding comments
are trimmed to the load-bearing why.
* shell: ec.encode batches 10 volumes by default
Encoding a whole collection as one batch means a late failure leaves
everything half-converted. Default -batchSize to 10 so each batch is
encoded, rebalanced, verified, and its originals deleted before the
next starts. -batchSize=0 keeps the all-at-once behavior. Batch
progress messages now print only when there is more than one batch,
so small runs read as before.
* shell: ec.decode decodes in batches, refreshing topology in between
ec.decode walked every volume off the single topology snapshot taken
at startup, which goes stale as earlier decodes move shards around and
create volumes. Decode 10 volumes per batch by default, re-collecting
the topology and rebuilding the free-space accounting between batches.
-batchSize=0 keeps the single-snapshot behavior.
Mounting freshly generated shards notifies the master asynchronously, while the post-encode rebalance plans from a fresh VolumeList snapshot; a snapshot that predates the delta shows zero shards, so the balance plans no moves and silently succeeds, and the original .dat is deleted with all shards on the generation host. Poll until every encoded volume reports a full shard set before balancing — across all disk types, since fresh shards register under the source volume's disk type, and scoped to the newest encode generation so an orphaned older generation neither satisfies the wait nor defeats the clump check. As defense in depth, refuse to delete originals when a volume's shards still sit on one node while another node has free EC slots.
* s3: accept raw semicolons in query strings
Go's url.ParseQuery drops any key=value pair containing a raw ';'. A
presigned PUT that signs content-type carries
X-Amz-SignedHeaders=content-type%3Bhost; when a client or proxy decodes
the %3B, the parameter vanished and the upload failed with
MissingFields, while AWS accepts the raw ';' as query data. Re-encode
it before routing so the pair survives parsing and signature
verification.
* iam, iceberg: recover raw-semicolon query pairs on the other listeners
The standalone IAM API verifies SigV4 with a canonical query recomputed
from the parsed query, and Iceberg REST warehouse/parent values may
legally contain ';'. Move the normalization middleware to util/http and
attach it to both routers.
A raw OIDC token sent as Authorization: Bearer was validated and mapped
to a role through the provider's roleMapping, then authorized against
the role's attached policies without ever consulting the role's trust
policy. A federated user could act as a role that
AssumeRoleWithWebIdentity would refuse to issue a session for with the
same token. Run the same trust-policy validation before returning the
principal on the bearer path.
* fix(filer.backup): stop silently dropping events on transient not-found
Under a write burst, filer.backup could consume a metadata event (advancing
the persisted offset) without replicating the file, with no error logged:
1. filersink CreateEntry/UpdateEntry swallowed replicateChunks errors
(glog.Warningf + return nil), so the offset advanced past entries that
were never written.
2. The manifest-chunk branch of replicateChunks resolved via LookupFileId
with no retry, unlike the data-chunk branch — transient lookup races
dropped exactly the large manifest-backed files while small
inline-content siblings landed.
3. isIgnorable404 matched "LookupFileId" / "volume id ... not found",
misclassifying those races as genuine source 404s at the backup layer.
Fix: on a replicateChunks failure the filer sink now skips only when the
live source has moved past the replayed version (deleted or strictly-newer
mtime) — lossless, a later event carries the current content — and
propagates otherwise so the event is retried. The manifest resolve retries
transient errors like the data-chunk path. isIgnorable404 is narrowed to
genuine 404s; non-filer sinks and the initial-snapshot walk, which relied
on the broad match as their only lossless-skip valve, now make the same
live-source decision (filersink.SourceSupersedes) instead of retrying
forever on a permanently gone volume.
Tests cover propagation of unconfirmed lookup failures, the narrowed 404
classification, and the supersession guards.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer.backup): derive supersession path with directory fallback
eventSourceSuperseded built the source path from NewParentPath alone.
Legacy metadata events (persisted by older filers) carry an empty
NewParentPath, so the probe looked up "/<name>", read the miss as
"source gone", and skipped a live file on a transient lookup error —
the silent drop this change is meant to eliminate.
Derive the path via MetadataEventTargetFullPath (the same directory
fallback genProcessFunction uses) and cover both event shapes with
TestEventSupersessionProbe_PathDerivation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filersink): retry manifest resolve only for transient errors, bounded when unverifiable
The manifest-resolve retry stopped only when hasSourceNewerVersion proved
the source moved past the replayed version, which wedged the sink in two
cases: incremental sinks use dated target keys that cannot map back to a
source path (supersession never provable), and permanent resolve errors
(corrupt manifest data, bad file ids) fail forever while the source entry
stays live.
Gate the retry instead: keep retrying only transient errors (volume-lookup
races, network interruptions), stop after a few attempts when supersession
cannot be checked, and propagate everything else immediately so the
configured metadata error policy applies (-disableErrorRetry included).
Propagation is lossless: filer.backup's fallback decides with the event's
real source key, and both filer.backup and filer.sync re-deliver the event
(RetryForeverOnError) without advancing the offset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer.backup): make error classifiers nil-safe
isIgnorable404, isSourceLookupError, and isTransientResolveError called
err.Error() without a nil guard. All current call sites pass a non-nil
error, but the guard is free and matches isRetryableNetworkError.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* admin: let waiting shell clients win the admin lock
The admin lock manager re-acquired the cluster shell lock immediately
after releasing it, while a waiting weed shell client only polls the
master once a second, so an operator could lose the race indefinitely.
Leave the lock free for slightly more than one poll interval before
re-acquiring, and once overlapping reference-counted holds have kept
the lock continuously held for over a minute, make new admin-side
acquires wait for a full release before piggybacking.
* admin/plugin: take the shell lock per detection and per job, not per batch
The default-lane scheduler acquired the cluster shell lock once and
held it across the whole pass: every due job type's detection plus all
of its dispatched jobs, each drained to completion. A manual weed
shell operation sharing that lock could stall behind the batch for up
to the extended execution window.
Hold the lock only while it protects something: around each detection
scan and around each dispatched job. Manual shell operations now wait
for at most one in-flight job. The admin UI detect+execute path stops
wrapping dispatch in an outer hold, since a nested acquire would
deadlock against the lock manager's fairness window; detection and
per-job dispatch take the lock themselves.
* admin/plugin: stop extending the dispatch window to the largest job estimate
When any proposal's estimated runtime exceeded the remaining
JobTypeMaxRuntime, the whole dispatch context was replaced with a
fresh one capped at eight hours, so a single balance backlog could
hold the default lane (and with it erasure_coding and vacuum
detection) for that long.
Keep the dispatch window at JobTypeMaxRuntime and instead detach each
started attempt onto its own estimated-runtime deadline. Large jobs
still get their full time once started; jobs not yet started when the
window closes are canceled and re-proposed by a later detection, so
sibling job types get a turn every window.
* worker/balance: re-check each planned move against the master before executing
A balance plan is computed at detection time, but the admin lock is
released between detection and execution, so a manual shell operation
can rearrange the volume in the gap. The task's own guards catch a
vanished source, but a target that gained a replica in the meantime
would be silently overwritten by VolumeCopy and the source delete
would then reduce the volume to a single copy.
Before executing each move, ask the master for the volume's current
locations (uncached) and skip the move if the volume has left the
source or the target already holds a replica. Skipped moves fail with
a stale-move error and the next detection replans them. Without master
addresses in the cluster context the check is skipped, preserving the
old behavior with older admins.
* admin: block re-acquire while the final lock release is in flight
Release dropped the manager mutex before calling ReleaseLock, so a
concurrent Acquire could see hold count zero and call RequestLock
while the locker still considered itself locked. That request no-ops,
leaving a hold with no live master lease. Track the in-flight release
and make Acquire wait for it.
Also normalize a nil release function from lock manager
implementations, and make the yield and fairness windows per-instance
fields so tests stop mutating globals.
* rust volume: accept odd-length needle id hex in file ids
Go formats the needle id with strconv.FormatUint and parses it back with
strconv.ParseUint, neither of which pads to an even number of hex digits.
hex::decode rejected such file ids with "Odd number of digits", so
volume.fsck could not purge orphans from a rust volume server. Parse the
needle id and cookie with from_str_radix, matching Go's ParseNeedleId
and ParseCookie.
* storage: emit even-length needle id hex in NeedleId.FileId
volume.fsck and volume.check_disk build purge file ids here with unpadded
FormatUint hex, while every other fid formatter strips whole leading zero
bytes. Pad to even length so the output matches the canonical fid format
and strict hex parsers accept it.
* s3: verify SigV4 against each plausible reverse-proxy host
A portless X-Forwarded-Host leaves the client's true port ambiguous: a
proxy that kept the Host header makes the backend Host port right, one
that rewrote it makes X-Forwarded-Port right, and a client on the
scheme's default port signed no port at all. The verifier bet on the
backend Host port whenever the hostnames matched, so nginx-style
$host/$server_port forwarding got SignatureDoesNotMatch whenever the
proxy and backend share a hostname. Try each plausible host value in
likelihood order instead of guessing one.
* s3: unbracket IPv6 X-Forwarded-Host before matching the request host
net.SplitHostPort strips brackets from the request host, so a bracketed
portless X-Forwarded-Host like [::1] never matched and lost its port
candidate.
* s3: cover unbracketed IPv6 forwarded-host candidates; compare with slices.Equal
TestSharedSnapshotConcurrentIntegrity writes as fast as one core can
marshal for two seconds, so its transient heap scales with host speed.
On linux/386 a fast runner walks the heap into the 4GB address-space
ceiling and the run dies with out of memory. Set a 1GB memory limit for
the duration of the test so the GC paces the writer instead of the
address space; the seal-and-recycle race being tested is unaffected.
* s3: keep listing when empty directories fill the listing window
doListFilerEntries issued a single ListEntries request per directory with
Limit = maxKeys+2. Entries that emit nothing - empty directories, the
.uploads folder, the marker echo - consume that window without consuming
maxKeys, so a bucket whose first window held only empty directories was
reported empty and not truncated, and a subdirectory whose window filled
up silently dropped the entries behind it. Keep requesting from the last
received entry until the quota is filled or a short window shows the
directory is exhausted.
* s3: test listing across windows of empty directories
The test filer client now honors StartFromFileName so repeated windows
advance like the real filer.
* s3: propagate the request context into directory listing RPCs
A disconnected client now cancels the ListEntries streams instead of
letting the listing keep issuing requests against the filer.
* s3: group per-directory listing parameters into a request struct
doListFilerEntries took ten positional parameters; call sites read as
string and bool soup. Wrap the per-directory arguments in
listDirectoryRequest so recursions and tests name what they pass.
* s3: group list request parameters into a struct
listFilerEntries took eight positional parameters ending in two bare
booleans. Wrap them in listObjectsRequest so the V1 and V2 handlers
name what they pass.
Move pickOneReplicaToCopyFrom, pickOneReplicaToDelete,
pickOneMisplacedVolume, and isMisplaced into weed/topology/balancer,
following satisfyReplicaPlacement. Selection functions take the shared
balancer.Replica shape and return indices, so the shell keeps its own
replica type; behavior is unchanged and covered by the existing shell
tests.
* volume.fix.replication: parallelize under-replicated volume copies
Fan out fixOneUnderReplicatedVolume up to -maxParallelization at a time.
Destination selection and free-slot accounting stay atomic behind a
scheduler mutex, so concurrent fixes see each other's reservations; a
failed copy returns its reserved slot. A per-server in-flight cap
(-maxParallelizationPerServer, default 1) keeps many simultaneous copies
from swamping a single destination server; when every eligible
destination is at the cap the fix waits for a slot instead of failing.
* volume.fix.replication: isolate the scheduler's location ordering
Clone allLocations before the per-volume fan-out so the scheduler's
re-sorting cannot alias the slice shared with the concurrent delete
phases, and make the test's per-iteration location copy explicit.
* s3: routed last-version delete removes the emptied .versions directory
The routed versioned delete (routedDeleteSpecificVersion) repoints the
latest pointer and deletes the version file, but unlike the lock-path
fallback (updateLatestVersionAfterDeletion) it never tears down the
.versions/ directory it just emptied. The residue keeps the key's read
path in the self-heal rescan loop: every GET of the deleted key logs
event=surfaced plus a GetObject error until the background
EmptyFolderCleaner gets to the directory — at least two minutes away on
its delay queue, and possibly never (the queue is in-memory, bounded,
and gated on the bucket's allow-empty-folders policy). Veeam's lock
arbitration probes deleted lock keys continuously, so those windows are
always open and the log spam is chronic.
ObjectMutation DELETE gains remove_empty_parent: after the child delete,
the filer best-effort removes the parent directory in the same locked
transaction. Non-recursive on purpose — a concurrent write that lands a
new version fails the removal instead of being lost with it. The routed
last-version delete sets it on the version-file DELETE, matching the
lock-path fallback's contract.
Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv
* s3: drain empty .versions residue on read heal and in s3.versions.audit
Directories already stranded by pre-teardown deletes (or dropped from
the EmptyFolderCleaner's bounded in-memory queue) previously re-entered
the self-heal rescan on every GET forever: the heal only cleared the
pointer and nothing ever removed the directory, and s3.versions.audit
counted the state as clean.
When the heal rescan finds no remaining version, remove the directory
outright (non-recursive, so orphan children still block and fall back to
the pointer clear) and log event=healed mode=empty_dir_removed; the next
GET takes the clean not-found path. The audit gains an empty category so
the residue is visible, and -heal removes such directories in bulk.
Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv
Align the usage percentage with s3.bucket.quota.enforce and the admin
UI, which measure quota against the live data size (size minus
un-vacuumed deleted bytes). Also print the logical size so both the
raw and live views stay visible.
* s3: keep empty-folder cleanup out of the multipart .uploads staging tree
The async EmptyFolderCleaner deleted <bucket>/.uploads once it looked
empty. A concurrent CreateMultipartUpload inserting its marker between
the cleaner's emptiness check and the bulk child delete had its row
wiped, so the upload silently vanished: ListMultipartUploads then omits
it and the following part/complete/copy calls fail. Skip the .uploads
subtree in both the queue and the delete path (including the eager
parent cascade); the multipart upload lifecycle owns it.
* s3: slice the bucket-relative path instead of reallocating it
* volume.balance: rank by physical disk usage
* volume.balance: keep -byDiskUsage ranking on one scale across the fleet
A server that does not report disk bytes ranks at whole volume-equivalents
while reporting servers stay below 1.0, so during a rolling upgrade the
balancer drains nearly empty old-build servers onto physically fuller ones.
Decide the scale once: rank by physical used percent only when every server
reports disk bytes, otherwise fall back to the data-size ranking for all.
Normalizing the fallback by maxVolumeCount instead would reintroduce the
over-configured-maxVolumeCount distortion this flag exists to avoid.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Drives the full path: AssumeRoleWithWebIdentity embeds the claims in the
session JWT, AuthenticateJWT restores them, and AuthorizeAction evaluates
ForAnyValue:StringEquals on jwt:groups / jwt:roles per bucket. The mock
OIDC provider now carries token claims through and surfaces roles,
matching the real provider.
* iam: surface OIDC groups into STS session request context for resource-policy ABAC
The groups claim was added to ExternalIdentity.Groups but excluded from Attributes
(processedClaims), so it never reached the session RequestContext. As a result
group membership was usable only in role trust policies (assume-time), not in
resource/permission policies (request-time) - so a single role could not scope
access by the caller's groups (aggregate ABAC). Surface Groups as a []string in
the request context; the string-condition evaluator already handles multi-valued
context keys, and the S3 middleware exposes it as jwt:groups.
* iam: also surface OIDC roles into the STS request context (companion to groups)
The 'roles' claim is excluded from the OIDC attributes by the same processedClaims
set that excluded 'groups', so it never reached the session request context and was
usable only via provider-configured role mapping - not a raw token roles claim. Add
ExternalIdentity.Roles, populate it from the token's roles claim, and surface it as a
[]string in the request context so resource policies can gate on jwt:roles, exactly
as the previous commit did for jwt:groups.
* log_buffer: share one window snapshot across all subscriber reads
Every in-memory read handed each subscriber a private pooled copy of the
window it wanted, so N subscribers reading the same data cost N copies of
up to 8MB each -- and slow consumers (grpc send backpressure) held those
copies live for their whole iteration. With hundreds of mount subscribers
that multiplied into gigabytes of live heap on the filer.
Share the bytes instead of copying per reader:
- Sealed windows get a lazily created GC-owned snapshot, made once by the
first reader and handed out zero-copy to the rest. The snapshot travels
with its window when SealBuffer shifts slots, so recycling the sealed
array never invalidates it.
- The current window keeps a shared snapshot of its append-only prefix
buf[:pos], extended on demand; each byte is copied once per window
(writer-rate-bound) instead of once per reader. At seal a fully
extended prefix becomes the sealed window's snapshot.
ReadFromBuffer now reports whether the returned buffer is a pooled copy
(flush path) or a shared view that must not be released; the read loops
only recycle pooled buffers.
With 200 subscribers consuming at grpc pace over sealed and current
windows, peak live heap drops from 5.2GB to 178MB.
* log_buffer: clear released read buffer so a panic cannot double-free it
The read loops release the previous iteration's pooled buffer and then
call ReadFromBuffer. If that call panicked before reassigning bytesBuf,
the deferred cleanup would put the same buffer into the pool a second
time, letting two future readers share one backing array. Nil the
pointer at the release site so the defer sees nothing to free.
* java client: fail cleanly when a chunk read returns short or empty data
readChunkView copied chunkView.size bytes out of the fetched chunk without
checking the fetched length, so an empty body (seen transiently right after an
append) threw an opaque IndexOutOfBoundsException. Retry empty chunk fetches and
bound the copy to the data actually returned.
* overflow-safe bounds check and null guard on fetched chunk data
LoadRemoteFile now takes dataFileAccessLock (so a live tier upload does
not race the heartbeat's DataBackend read). But load() also runs it, and
CommitCompact calls load() while already holding that lock, so reloading
a remote-tiered volume during a compaction commit re-enters the
non-reentrant lock and deadlocks.
Split the locked bodies out: swapDataBackendLocked and loadRemoteFileLocked
assume the caller holds dataFileAccessLock. load() uses loadRemoteFileLocked;
the public LoadRemoteFile keeps taking the lock for the live tier-upload
handler that does not hold it.
* log_buffer: read only ts_ns during metadata-read binary search
readTs ran a full proto.Unmarshal of each LogEntry just to compare
timestamps during ReadFromBuffer's binary search, allocating fresh
slices for the data and key byte fields on every probe. Under
metadata-subscription fan-out this decode churn dominated allocations
and drove heavy GC, inflating filer RSS well past the live heap.
Scan the wire format for ts_ns (field 1) instead and skip the data/key
payloads without copying them. readTs is now zero-alloc; the binary
search no longer touches the event payloads at all.
* log_buffer: alias event payloads instead of copying them on delivery
The subscribe read loops decoded each LogEntry with a full proto.Unmarshal,
allocating fresh slices for the data and key byte fields on every entry
(protobuf consumeBytesNoZero). When many metadata subscribers each re-read
the in-memory log window under a reconnect storm, that per-entry copy was a
large share of allocation churn -- and for entries a subscriber filters out
by prefix, the copied payload was never even looked at.
Decode with a wire scan that points data/key at sub-slices of the source
buffer instead of copying. The aliased slices are valid only for the
duration of the eachLogEntryFn callback; every current subscriber either
re-decodes the event into its own struct or hands it to a synchronous grpc
Send, so none retain them. Per delivered entry the loop decode drops from
176 B / 2 allocs to zero.
VolumeTierMoveDatFromRemote downloads the .dat, trims the .vif, and swaps
the data backend to the local file, but left hasRemoteFile set. The
volume.tier.download command masks this by unmounting and remounting
right after, which reloads the flag from the trimmed .vif — but in the
window before the remount the in-memory flag is wrong: doDeleteRequest
would skip appending the tombstone to the freshly local .dat, and the
phantom-.dat guard stays disabled.
Give SwapDataBackend a hasRemoteFile argument so the backend swap and the
flag move together under one lock, and route both tier directions through
it: the tier-down download passes false, LoadRemoteFile passes true. The
flag can no longer disagree with the live backend.
volume: keep tier-uploaded volume reporting to master
A live volume.tier.upload removes the local .dat and swaps the data
backend to remote, but v.hasRemoteFile was only ever set when a volume
is loaded from disk, so the running volume kept it false. The phantom
.dat guard then saw fileCount>0, !HasRemoteFile, and a missing .dat, and
stopped reporting the volume to the master. The volume vanished from the
topology even though the upload succeeded and the data was in cloud
storage.
Set hasRemoteFile in LoadRemoteFile, the single point where a volume's
backend becomes remote, so it is true both on disk-scan load and after
an in-process tier upload. Route the backend reassignment through
SwapDataBackend so it happens under dataFileAccessLock, closing the old
backend and never racing the heartbeat's concurrent DataBackend read.
Make the field atomic since the heartbeat now reads it concurrently with
the tier-upload handler that writes it.
* java: HTTP Basic Auth for filer behind an Nginx reverse proxy
Attach an "Authorization: Basic" header to the filer gRPC channel through a
ClientInterceptor and to the volume read/write HTTP requests, so the Java
client can reach SeaweedFS fronted by Nginx with auth_basic enabled.
Credentials come from a [basic_auth] section in security.toml or from
FilerSecurityContext.setBasicAuth(). On writes the volume JWT still owns the
Authorization header when present, since the two cannot share it.
* java: precompute Basic Auth header and warn on plaintext channel
Cache the Base64 Authorization value once instead of re-encoding on every
chunk read/write, and log a one-time warning when Basic Auth rides a
plaintext gRPC channel where the credentials would travel in cleartext.
* s3: fail over routed object writes when the owner filer is unreachable
A routed object write (multipart completion, PUT, delete, versioned
finalize, metadata replace) dialed the ring-selected owner filer
directly with no failover. After a filer restarts onto a new address the
lock ring can still name the old one, so every routed write hangs on the
dead address until the gateway is restarted; CompleteMultipartUpload in
particular exceeds client timeouts.
Route the transaction through withFilerClientFailover, skipping an owner
that recently failed, so a live filer forwards it to the real owner by
route_key. Mirrors the read path's getObjectEntryRoutedByKey.
* s3: fail over bucket-config writes when the owner filer is unreachable
patchBucketEntry dialed the bucket's ring owner directly, so a restarted
filer's stale ring address hung every bucket-config write (versioning,
lifecycle, object lock, ownership, ACL, policy, CORS) - the same failure
as routed object writes. Route it through objectTxnOnFiler so it skips an
unreachable owner and a live filer forwards by route_key.
* Optimize s3api encodePath to eliminate O(n^2) allocations
encodePath built its result via string concatenation in a loop, which is
O(n^2) in allocations. For non-ASCII (e.g. Chinese) runes it additionally
called make + hex.EncodeToString + strings.ToUpper per byte (~10 allocations
per character). Under high QPS with long non-ASCII object keys this produced
a very high allocation rate, frequent GC and long GC pauses, causing S3
request latency spikes.
Replace with a preallocated strings.Builder, a manual hex lookup table, and a
zero-allocation fast path. EncodePath now delegates to encodePath to remove
the duplicated implementation. Output is byte-for-byte identical, verified by
TestEncodePath and TestEncodePathEqual.
Benchmark (long Chinese path): 261 -> 1 allocs/op, 35810 -> 480 B/op, 7.5x faster.
Load test (50M calls): 212x fewer allocations, 76x fewer GC cycles.
* s3api: drop regexp from encodePath fast path
The unreserved-character scan already decides whether any byte needs
encoding, so reservedObjectNames.MatchString was a redundant second pass
that also ran the RE2 engine on every authenticated request. Rely on the
scan alone; output is unchanged. The ASCII fast path drops from ~188ns to
~11ns per call.
---------
Co-authored-by: LiuDoge <liudoge@LiuDogedeMacBook-Air.local>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: require JWT authorization on TUS upload endpoints
filerHandler and readonlyFilerHandler run every request through
maybeCheckJwtAuthorization, but the TUS routes were registered only
behind filerGuard.WhiteList, which is a pass-through for the filer (its
guard is built with an empty whitelist), and no TUS handler called the
JWT check. So with jwt.filer_signing.key set, normal PUT/POST/DELETE
were authorized while the TUS endpoints were not.
Run the same check in tusHandler before routing: HEAD uses the read key,
POST/PATCH/DELETE use the write key, and creation scopes a
prefix-restricted token against the resolved target path. OPTIONS stays
open for capability discovery.
* filer: enforce read-only and WORM rules on TUS completion
completeTusUpload wrote the final entry with CreateEntry directly,
skipping the read-only and WORM checks the normal write path applies.
Reject completion when the target prefix is read-only or the existing
entry at the target is WORM-enforced.
* filer: align TUS write path with the normal write path
Resolve the create target with a guaranteed leading slash so a
prefix-restricted token and the stored path stay absolute even if
TusBasePath were set with a trailing slash. Reject read-only prefixes at
session creation before any chunk is written, and map read-only and WORM
rejections at completion to 507 and 403 instead of a generic 500.
* filer: cover method-restricted TUS tokens in the auth test
With replication 001 and one node down, ~2/3 of volumes have their
replica on the dead node. The primary write lands locally but the
replica forward fails, so the volume returns 500 "failed to write to
replicas". #9744 made that upload fail fast so the client re-assigns,
but the client retry never fired: the reassign gate matched a fixed
list of error substrings that didn't include this one. The mount
surfaced I/O error and dropped the chunk, leaving missing lines and
null-byte gaps in an append workload while a node rebooted.
Decide reassignment by HTTP status instead of matching message text.
On the write path a volume server only 5xxs on a ReplicatedWrite
failure (local disk, replica peer down, under-replication) — all of
which a different volume dodges — so any 5xx reassigns; a no-response
transport failure (the assigned target itself is down) reassigns too;
a 4xx is a genuine client error and is surfaced. doUploadData tags its
errors with the response status via uploadStatusError, and the gate
moves from util.MultiRetry(errList) to util.RetryOnError(predicate).
This drops the fragile substring list (and the message-prefix
constants and guard test it needed); store_replicate.go is untouched.
* filer: keep .system internal folder in the default SQL table
Bucket-table SQL stores read the first path segment under /buckets as a
bucket name. The ListBuckets owner index lives at /buckets/.system/..., so
every write there hit isValidBucket(".system") == false and returned
"invalid bucket name .system", flooding the filer log on the postgres/mysql
backends. Route dot-prefixed internal folders to the default table by their
full path, like any other non-bucket entry.
* filer: keep .system internal folder in the default leveldb3 DB
Same guard as the SQL stores: leveldb3 would otherwise open a separate DB
for the .system owner-index folder instead of keeping it in the default DB.
* filer: keep .system internal folder under the default ydb prefix
Skips a DescribeTable round trip per operation on the .system owner-index
path, which never resolves to a real bucket table.
* filer: keep .system internal folder in the default arangodb collection
Avoids creating a stray collection for the .system owner-index folder.
* seaweed-volume: async, buffered writes in VolumeEcShardsCopy
The EC-shards-copy RPC handler wrote each streamed chunk to disk with a
synchronous std::fs::File::write_all inside the async handler, blocking a
Tokio worker thread for the duration of every write — noticeable for a
large .ecx on a slow or busy disk.
Factor the five near-identical receive-and-write loops (.ec shards, .ecx,
.ecj, .vif, .ecsum) into drain_copy_stream_to_file, which uses tokio::fs +
BufWriter for async, buffered I/O. Behavior is otherwise unchanged: the
.ecj append mode, the .ecsum byte count and 0-byte-file cleanup, and all
error messages are preserved.
Claude-Session: https://claude.ai/code/session_01Ny5Rt1ph9VWeKmfY936GtF
* seaweed-volume: remove partial copy target on error in EC-shards-copy
Follow-up: drain_copy_stream_to_file now deletes the destination file on
any recv/write/flush error, so a failed VolumeEcShardsCopy no longer leaves
a truncated .ecNN/.ecx/.ecj/.vif/.ecsum on disk for a later reader to trip
on. Matches receive_file / the Go volume server. Best-effort cleanup; the
original stream error is still returned.
Claude-Session: https://claude.ai/code/session_01Ny5Rt1ph9VWeKmfY936GtF
CRC64NVME multipart objects now emit a full-object checksum instead of a composite base64-N value, matching AWS (CRC64NVME supports full-object only). Adds x-amz-checksum-type handling (COMPOSITE/FULL_OBJECT) for CRC32/CRC32C/CRC64NVME via CRC combination, resolved at CreateMultipartUpload and applied at completion.
* cassandra2: default connection_timeout_millisecond so env-var configs keep the 600ms timeout
The scaffold documents a 600ms default, but the value has no SetDefault. A
config supplied purely through env vars (or a minimal toml) that omits the
key read back 0, silently overriding the client timeout with no bound.
* ydb: default the table path prefix so env-var configs keep the seaweedfs sub-path
The scaffold documents prefix = "seaweedfs", but with no SetDefault a config
that omits it lands tables at the database root instead of under seaweedfs/.
* postgres2: default the filemeta CREATE TABLE when createTable is unset
An empty createTable rendered through fmt.Sprintf produced
%!(EXTRA string=filemeta), which Postgres rejected with a syntax error at
init. Fall back to a working template so a minimal config bootstraps.
* mysql2: default the filemeta CREATE TABLE when createTable is unset
Same empty-template failure the postgres2 path had: an unset createTable
rendered to %!(EXTRA string=filemeta) and MySQL rejected it at init.
Fall back to a working template.
aws-lc-rs and ring both get linked transitively, so rustls can't
auto-select a crypto provider and tonic's client TLS panics the moment
the volume server dials a master over TLS. Install aws-lc-rs as the
process default in main(), matching the provider the server config
already uses.
* shell: add -delete option to remote.copy.local
Add a -delete flag to remote.copy.local that removes files and
directories from remote storage when they no longer exist locally,
similar to rsync --delete. This makes the command usable for
scheduled one-shot backups that also propagate local deletions.
- -include/-exclude patterns also limit which remote files are deleted
- size/age filters only apply to copying, since remote entries have no
local attributes to filter on
- orphaned remote directories are removed after their contents,
deepest first, and only when no name filter is set (a recursive
RemoveDirectory could otherwise remove intentionally kept files)
- deletion is skipped entirely if any copy failed
- -dryRun shows DELETE lines for review before committing to anything
Fixesseaweedfs/seaweedfs#8609
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* shell: fix remote.copy.local -delete deleting files outside -dir
Traverse lists remote objects by key prefix with no delimiter, so a -dir
pointing at a subdirectory also matches siblings that merely share its
name prefix (foo -> foobar). Those are not under the local traversal
root, so -delete treated them as extraneous and removed them. Scope
deletion candidates to paths under dirToCopy.
Also drop the directory-removal path: RemoveDirectory is a no-op on every
backend and Traverse never emits directory entries, so it only ever
printed success for work it never did.
---------
Co-authored-by: Jason Lin <jason@jtx.com.tw>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
A read or a replicated write to a volume server that is TCP-reachable but not
answering -- one still loading its volumes after a restart, or reached over a
stale keep-alive to a container that came back on a new IP -- blocked forever:
the shared HTTP transport had a dial timeout but no response timeout.
Add ResponseHeaderTimeout so a chunk read fails over to another replica and a
replicated write fails fast for the client to retry, and IdleConnTimeout so
pooled connections to a departed server are evicted instead of reused.
* s3: answer application/x-directory for a directory without a stored mime
A real directory reached via a trailing-slash GET/HEAD answered the
octet-stream default when it had no stored mime, so Hadoop-style S3
filesystems (flink-s3-fs-presto and friends) classified the path as a
0-byte file instead of a directory and then failed reading it as one.
Answer application/x-directory, the marker type those clients probe
for. Stored mimes still echo verbatim, and a file promoted to a
directory keeps octet-stream for its data.
* s3: 404 GET and HEAD on a bare directory path consistently
A directory with no object data of its own answered differently per
path: plain GET gave an empty 200, ranged GET and non-versioned HEAD
gave 404, and on versioned buckets the null-version fallback adopted
the filer directory as a 0-byte object and answered 200. Clients that
probe HEAD-then-GET took the 200s at face value, treated the path as
an empty file, and never fell back to LIST-based directory discovery.
Answer 404 for a bare directory path everywhere, which is what AWS
returns for a prefix. A file promoted to a directory keeps its data
and stays retrievable.
* master: re-create a vacuum-committed volume the index has lost
SetVolumeAvailable dereferenced vid2location[vid] with no nil check. A
disconnect during a long vacuum can drop a single-replica volume from the
lookup index while it stays on the node; the commit then panics the master
instead of re-registering it. Re-create the entry, and seed its size tracking
so assigns are counted right away rather than after the next heartbeat.
* master: re-register a vacuumed volume the index lost on mark-writable
The maintenance worker re-enables a volume by marking it writable after the
vacuum commit, but VolumeMarkReadonly only updated nodes the lookup index
already knew. If a disconnect race dropped the volume from the index during the
vacuum, that path was a no-op and the volume stayed "not found" until the next
full heartbeat healed it. Re-register it from the node that still holds it.
* admin: move volume-server read JWT helper into dash
The Iceberg data preview page needs the same per-fileId read token the
file browser uses when streaming chunks from volume servers.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: add Iceberg table data preview page
The admin UI browses the Iceberg catalog down to table details but not
the data itself. Add a Browse Data page per table that walks the
selected snapshot's manifests and shows sample rows from its Parquet
data files, plus the data file list with per-file preview, a snapshot
switcher, and a row limit selector.
Rows are read through a ranged ReaderAt over stream-content so only
the Parquet footer and needed pages are fetched, with the volume read
JWT applied when configured. Iceberg locations resolve into /buckets
with traversal guards, and the file parameter must match a
manifest-listed data file. Snapshots with delete files get a warning
that raw rows are shown.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: integration test for Iceberg catalog and data preview pages
Starts a weed mini cluster with the admin UI, creates a table bucket,
namespace, and tables via the S3 Tables manager, uploads real Parquet
files via S3, writes manifests and snapshots with iceberg-go, and
asserts on the rendered pages: catalog browsing, table details,
current and historical snapshot previews, per-file preview, row
limits, unknown snapshot and file errors, and a metadata-less table.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: write Iceberg preview chunk reads straight into the caller slice
ReadAt wrapped the caller's buffer in a bytes.Buffer, which would
silently allocate a fresh backing array and drop bytes if it ever grew.
Copy directly into the destination slice and reject negative offsets so
the ReaderAt contract holds.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: link to snapshot history when the preview switcher truncates
The snapshot switcher caps at 25 entries; add a trailing item pointing
at the table details page so older snapshots stay reachable.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* test: hoist mini cluster context assignment out of the goroutine
Set MiniClusterCtx before launching the cluster goroutine and clear it
in stop(), so the assignment is not buried in the command loop.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* s3: enforce bucket quota on logical size, not un-vacuumed physical size
A bucket full of deleted/overwritten objects awaiting vacuum went
read-only while its live data stayed under quota, because enforcement
used the raw single-copy volume size with garbage included. Subtract
DeletedByteCount via a LogicalSize() helper in the auto-enforce loop,
the s3.bucket.quota.enforce command, and the bucket_size_bytes metric
(labeled logical but counting garbage too). Deleting objects now
relieves quota immediately and enforcement matches the UI usage figure.
* admin: surface bucket read-only state in the S3 buckets UI
Read the read-only flag quota enforcement writes to filer.conf and show
it as a badge in the bucket list and a Status row in the details modal,
so an operator can see why writes are being rejected.
* feat(k8s): add Traefik IngressRouteTCP for gRPC with TLS passthrough
Re-introduce Traefik support for the gRPC filer ingress that was
lost when the original ingress PR was merged. Previous attempts to
make the chart controller-agnostic using Ingress + ServersTransport
+ TLSOption CRDs were fragile — they required 2 separate services
(HTTP and gRPC), still failed with connection resets, and forced
Traefik to terminate and re-encrypt TLS traffic.
This approach uses a single IngressRouteTCP CRD with TLS passthrough
when enableSecurity is true, keeping the TLS stream intact. No
ServersTransport, no TLSOption, no service annotations, no values.yaml
structure changes. Fully backward compatible.
Refs: seaweedfs/seaweedfs#10205
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* refactor(k8s): only render standard gRPC Ingress when className is not Traefik
When className contains 'traefik', the IngressRouteTCP is the only
source of truth. The standard Kubernetes Ingress becomes superfluous
and potentially confusing for debugging.
Now:
- className: traefik → only IngressRouteTCP
- className: nginx/contour/... → only standard Ingress
- className: "" (default) → neither
No values.yaml changes. Fully backward compatible.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* k8s: fix Traefik gRPC IngressRouteTCP for non-TLS and all-in-one modes
A non-TLS TCP router can only match HostSNI(`*`), so the default
enableSecurity=false path never matched. Use HostSNI(`*`) when security
is off and keep host-based SNI for TLS passthrough.
Route to the all-in-one service in all-in-one mode via the same ternary
the standard ingress uses; the hardcoded filer-client service is absent
when filer.enabled is false.
Also require grpc.enabled to render, align labels with the sibling
ingress, and put the comments in English.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The owner index read the raw filer entry off BucketConfig, which no
longer carries one. Cache Crtime alongside IdentityId, and pass the
owner id rather than the entry through the ListBuckets visibility
checks so the scan and granted-bucket paths share one implementation.
* s3: paginate ListBuckets with max-buckets, continuation-token, and prefix
ListBuckets buffered every bucket entry into one slice and one XML body,
which falls over with very large bucket counts. Page through the filer
listing instead, cap each response at 10000 buckets like AWS, and honor
max-buckets, prefix, and an opaque keyset continuation-token.
* s3: maintain a bucket owner index under /buckets/.system/owners
Map each bucket owner to its buckets as zero-length entries at
/buckets/.system/owners/<owner>/<bucket>, with Crtime mirroring the
bucket's creation time. The bucket handlers write the index
synchronously, the /buckets metadata subscription reconciles changes
made elsewhere (weed shell, other gateways, direct filer operations),
and a startup backfill indexes pre-existing buckets before writing a
ready marker. Owner names are path-escaped so no identity name can
escape the index directory.
* s3: serve ListBuckets from the bucket owner index
Once the owner index is ready, non-admin identities list their owned
buckets straight from it, merged with any buckets their legacy actions
name explicitly, so ListBuckets costs O(own buckets) instead of a scan
of the global /buckets directory. Admins, identities with a bare List
grant or wildcard action patterns, and policy-authorized identities
whose grants cannot be enumerated keep the paged scan; policy-routed
identities get their owned buckets, matching AWS ListBuckets returning
only the caller's buckets.
* s3: keep dot-prefixed names under /buckets out of bucket surfaces
Dot-prefixed entries (.system) can never be valid bucket names, so
refuse to resolve them as buckets and skip them in the shell bucket
listing, matching what ListBuckets and the admin UI already do.
* test: cover ListBuckets pagination and the owner index end to end
* s3: fail closed on a nil identity when routing ListBuckets
* s3: decide the IAM authorization mechanism in one place
VerifyActionPermission and the ListBuckets owner-index routing each
re-derived the session-token / attached-policy / legacy-actions split;
extract the decision so the two cannot drift.
* s3: heal the owner index on concurrent bucket recreation too
The mkdir-lost-the-race path answers BucketAlreadyOwnedByYou just like
the up-front existence check, so give it the same index repair.
* s3: drop owner-index records for buckets deleted during backfill
A bucket removed between the backfill reading its page and writing the
index record became a permanent phantom in its owner's listing: the
delete's own cleanup ran before the record existed. After indexing each
page, re-list the same name range and remove records whose bucket is
gone; deletes landing after the re-list find the record and remove it
themselves.
* s3: add ContinuationToken and Prefix to the ListBuckets schema
Keep AmazonS3.xsd aligned with the generated ListAllMyBucketsResult so
a regeneration does not drop the pagination fields.
* s3: stop warming every bucket's config at startup
Listing all buckets in BucketRegistry.init() made S3 gateway startup
O(buckets) and pinned every bucket's metadata and config resident, which
does not scale past a few hundred thousand buckets. Both caches already
have lazy miss paths, so load on first access instead and let the
metadata subscription refresh only entries already resident; cold
buckets cost one filer round-trip on their first request.
* s3: bound the per-bucket caches with LRU eviction
The bucket config cache, bucket registry, and their negative caches were
plain maps that only ever grew: the config cache TTL made Get miss but
never evicted the entry, and the not-found sets grew on every probe of a
nonexistent bucket name. Cap all four at 65536 entries with LRU eviction
so a gateway keeps its hot working set and evicted buckets reload from
the filer on next access.
* s3: cache parsed bucket config instead of the full filer entry
Each cached BucketConfig retained the whole bucket entry (extended
attribute map plus raw content bytes) alongside the fields parsed from
it, roughly doubling per-bucket cache cost and keeping data the read
path never looks at. Parse everything up front in
newBucketConfigFromEntry - now also the creator identity, tags,
encryption config, and stored lifecycle XML - and drop the entry.
updateBucketConfig now reads the entry fresh from the filer and diffs
the mapped extended attributes against it, so the patch is computed
against current state instead of a cached copy; the config clone
helpers that existed for that path go away.
* s3: dedup cold bucket-registry loads per bucket
The registry's notFound lock doubled as the load serializer, holding one
global mutex across the filer round-trip so first-touch requests for
different buckets queued behind each other; the cache fill also happened
after the lock was released, so two concurrent misses for the same
bucket could both reach the filer. Replace it with a singleflight per
bucket that fills the cache inside the flight: different buckets load
concurrently, the same bucket loads once.
glog flags (-v, -logtostderr) are registered on weed's global flagset,
so passing them after the subcommand name kills the process at flag
parsing: flag provided but not defined: -logtostderr. The old stat-based
mount readiness probe masked this — TestWriteBufferCap silently ran
against the bare local directory and passed. The device-ID readiness
check now surfaces the dead mount as a not-ready timeout.
Move glog flags into MountGlobalOptions, emitted before the subcommand,
and do the same for the EnableDebug verbosity flag on mini and mount.
* util: don't let the activity timeout clobber externally-set conn deadlines
util.Conn extended the connection deadline at the start of every
Read/Write. net/http's server also sets deadlines directly on the same
conn - abortPendingRead sets one in the past to interrupt the pending
background read after each response. The activity extension raced with
and silently overwrote that interrupt, leaving the read blocked (and the
server's conn.serve goroutine stuck in abortPendingRead) until the full
-idleTimeout (default 30s) expired.
Wedged connections count as active, so the volume server's graceful HTTP
drain waited out its whole 30s StopTimeout on shutdown - observed as
weed mini taking ~30s to exit in the FUSE integration tests after any
filer->volume traffic.
Track externally-set deadlines, suspend the activity extension while one
is in force, and serialize deadline updates with a mutex. Activity still
extends both directions at once: a long write-only response must keep
the read deadline alive too, or the server's background read would time
out and cancel the in-flight request.
* util: extend read/write deadlines independently when one side is external
A server-configured WriteTimeout keeps an external write deadline in
force for the whole request, which previously suspended the activity
extension entirely - leaving the read deadline stale from before the
request and letting net/http's background read time out mid-response.
Extend each direction independently instead.
* fix(master): let the growth initiator wait for the growth it triggered
The growth-in-flight shed also fired on the request that initiated the
growth: it sets the pending flag right before the shed check, so a
cold-start assign enqueued growth and immediately failed itself with
"volume growth in progress". With no concurrent assigns around to pick
up the freshly grown volume, a single writer against an empty cluster
never completes a write despite ample free space.
Claim the pending flag with a compare-and-swap so exactly one request
becomes the initiator, triggering growth at most once, and let it wait
for that growth to land. Everyone else still sheds retryably instead of
pinning a goroutine: followers behind an in-flight growth, an initiator
whose growth concluded without yielding a writable volume, and an
initiator whose growth outlives the 10s wait budget, which previously
surfaced a non-retryable error (gRPC Unknown, HTTP 406) even though a
retry would have succeeded moments later.
* fix(master): stop assign waits when the request is cancelled
The assign retry loops slept through client cancellation, keeping a
goroutine spinning for the rest of the 10s budget after the caller had
gone; StreamAssign also ran assigns on a background context detached
from the stream. Wait on the request context and pass the stream
context through.
* topology: drop the unconditional grow-request setter
Growth is only claimed through AddGrowRequestIfAbsent's compare-and-swap
now; keeping the raw Store(true) around invites the check-then-set race
back.
* test: cover cold-start first write with a real cluster
Boot a fresh master plus three empty volume servers and require the very
first assign - HTTP and gRPC, each on a cold volume layout, no client
retries - to complete a write. The assign that triggers volume growth
must wait for it rather than answering "volume growth in progress";
unit tests stub the topology, so only a real cluster exercises the
assign-grow-wait path end to end.
* test: keep mini-allocated ports below the ephemeral floor
Allocated ports could land in 32768-55000, so a transient outbound dial
during mini startup (volume->master gRPC, etc.) could grab an allocated
port as its source port before the filer bound it, failing with
"bind: address already in use". Cap the range so port+GrpcPortOffset
stays under 32768.
* test: derive mini port cap from the ephemeral floor constant
Name the 32768 floor once and compute miniPortMax as floor-GrpcPortOffset
so the cap tracks the offset; reuse the constant in the regression test.
waitForMount probed the mount point with stat+ReadDir, which a bare local
directory also passes, so Setup could return before the weed mount process
finished mounting. Tests then wrote to the local disk underneath the mount
point; when the mount activated it shadowed those files, producing the
intermittent TestConcurrentFileOperations/ConcurrentReadWrite ENOENT with
an empty ReadDir. Require the mount point's device ID to differ from its
parent's before reporting ready.
* s3: surface multipart-list store errors instead of masking them
listMultipartUploads masked a filer/metadata-store list error as an empty
200 response, and listObjectParts masked it as NoSuchUpload. A client
resuming a multipart upload (the Docker Registry S3 driver resolves an
in-progress upload via ListMultipartUploads, then ListParts) reads either
as "upload gone" and fails the upload permanently with a non-retryable
error. Return ErrInternalError so a transient store error stays retryable,
keeping ErrNotFound as an empty list / NoSuchUpload respectively.
* s3: return retryable error for transient CopyObject source lookups
Resolving the copy source is reported as InvalidArgument ("Copy Source must
mention the source bucket and key") whenever the lookup returns any error,
including a transient store error. Clients don't retry a 400, so a resumable
blob commit fails permanently. Keep the client error for a missing, invalid,
directory, or delete-marker source; map a transient store error to
ErrInternalError.
* s3: mark versioned lookup miss with the not-found sentinel
recoverLatestVersionWithoutPointer's terminal miss (a .versions directory
with no pointer, no versions, and no null object) returned a plain error,
so errors.Is-based callers could not tell this designed NoSuchKey state
from a store failure and reported it as an internal error.
* s3: reject a delete-marker copy source with NoSuchKey
getLatestObjectVersion returns the delete-marker entry when the latest
version is a delete marker, and the copy handlers copied its empty stub
into the destination. Every other handler detects ExtDeleteMarkerKey and
answers NoSuchKey; do the same for CopyObject and UploadPartCopy.
* s3: align copy-source not-found detection with the grpc status idiom
The lookup path canonicalizes text-form not-found into the sentinel in
filer_pb.LookupEntry and wraps with %w after that, so matching sentinel
text here was unreachable -- and risky, since a store error whose message
merely mentions the sentinel would be downgraded to a terminal 400.
Match the raw grpc NotFound code instead, like the other version-lookup
sites, and give transient filer errors the 503 that the upload-entry
lookup in this file already returns.
* s3: keep source-bucket versioning lookup errors retryable in copy
CopyObject and UploadPartCopy mapped any source-bucket versioning-state
lookup error to a terminal InvalidCopySource, including transient store
errors; getVersioningState signals a missing bucket with the not-found
sentinel, so split on that and let real store errors surface as 500, the
same mapping the destination-bucket lookup already uses.
* s3: match grpc-transported not-found in multipart list error paths
The list client path returns raw grpc status errors without the sentinel
reconstruction that lookups get in filer_pb.LookupEntry, so errors.Is on
the not-found sentinel never matched a store-reported missing directory;
match the sentinel text as well via a shared helper. The common missing-
directory case still lists as empty with no error and is unaffected.
* s3: complete and abort multipart surface store errors
prepareMultipartCompletionState mapped any upload-directory list or
lookup error to NoSuchUpload, so a transient store error at completion
time made the client discard a fully-uploaded object as gone. Split on
not-found like the sibling listing paths. abortMultipartUpload did the
same on s3a.exists, whose errors are never not-found (filer_pb.Exists
reports that as false with no error) -- a store failure there answered
NoSuchUpload and silently leaked the uploaded parts.
* s3: reject part numbers below 1 in UploadPartCopy
Only the upper bound was checked, unlike PutObjectPart; partNumber=0
passed the route regex and validation and wrote an out-of-range
0000_copy.part into the upload directory.
* filer: stream offloaded metadata-log entries instead of buffering whole files
The client metadata-chunks read path (ReadLogFileRefs, used by the meta
aggregator to consume peer filers and by mounts) decoded every entry of a
log file into a slice before handing it to the consumer, and prefetched the
next whole file the same way. Peak memory scaled with log-file size: under
heavy concurrent writes the per-event chunk lists grow and minute-files reach
hundreds of MB to GBs, so a filer aggregating a few peers held many GBs of
decoded entries at once (heap dominated by readLogFileEntries ->
consumeBytesNoZero) and OOMed.
Stream entries through a bounded channel: a producer decodes one entry at a
time and the next file's read overlaps processing via the channel buffer, so
peak memory is bounded by the channel depth rather than O(file size). In a
synthetic replay peak live heap dropped from ~1.3x the file size to a flat
few MB regardless of file size.
* filer: tighten offload replay tests
Share one ordered-replay assertion between the merge-order and single-filer
tests, assert the callback's own error is what propagates, and drop atomic
counters from callbacks that run on a single goroutine.
* filer: abort offloaded log replay promptly instead of joining wedged readers
Collapse the single-filer fast path into the merged reader: it was a second
copy of the producer/stop lifecycle with its own subtler synchronization, and
a one-stream merge does the same job.
On abort (processing error or a fatal read error from one filer), the
consumer used to drain channels and join every producer. A producer blocked
in an uncancellable chunk read cannot observe stop until that read returns,
so an abort could stall the caller's retry loop behind a dead volume-server
connection. Closing stop is now the only cleanup: producers check it at every
send and file boundary and exit on their own, and the merge loop's blocking
receives also escape on stop. Producers also check stop before opening each
file, so an aborted replay no longer keeps reading remaining files whose
entries never reach the channel.
A mid-file chunk-not-found still skips to the next file, but the log line now
reports how many entries were delivered first instead of pretending the whole
file was skipped; the redundant error log before setFatal is gone since the
error propagates to callers that already log it.
* filer: cap offloaded log entry allocation against corrupt size prefix
A garbage 4-byte size prefix (torn chunk or stream desync) drove
make([]byte, size) up to 4GiB per entry. Reject sizes above the same 1GiB
bound the filer-side log readers enforce.
* feat(k8s): add HTTP + gRPC Ingress templates for filer
Add HTTP and gRPC Ingress templates for the filer component in both
standalone and all-in-one modes. The HTTP ingress handles REST API
traffic, the gRPC ingress exposes the gRPC endpoint with proper
annotations for nginx and Traefik.
Additionally add Traefik IngressRouteTCP for mTLS filer gRPC passthrough.
When the filer has mTLS enabled, the standard HTTP Ingress terminates TLS
at the ingress level which conflicts with the filer's mutual-TLS requirement.
IngressRouteTCP forwards raw TCP with tls.passthrough: true so the TLS
negotiation happens directly between client and filer.
Refs: PR #10035 (original fix-grpc-filer)
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* feat(k8s): restructure filer ingress into ingresses.{http,grpc}
Split the single filer ingress value into http and grpc sub-structures
so the HTTP Ingress and gRPC Ingress templates each have their own
configuration.
* k8s: document nginx ssl-passthrough for end-to-end mTLS gRPC
The filer's mTLS gRPC needs the TLS stream to reach the filer intact,
which an L7 Ingress can't do when it terminates TLS. Document the
ingress-nginx ssl-passthrough annotation on the gRPC ingress so the
whole chart stays on the standard Ingress kind, no controller-specific
CRD required.
* k8s: align filer ingress with the volume/admin ingress pattern
Only render ingressClassName when a class is set (an empty value opts out
of the cluster's default IngressClass), fall back to the
kubernetes.io/ingress.class annotation on k8s <1.18, version-gate
pathType, and quote the host so wildcard hosts stay valid YAML.
* k8s: route the filer gRPC ingress at / with Prefix
gRPC methods are called at /<package>.<Service>/<Method>; the HTTP UI
regex path never matches them, so gRPC requests would 404.
---------
Co-authored-by: MorezMartin <martin.morez@morez.org>
Unmount, and optionally delete, EC volume shards from the shell, so broken
or over-replicated shards can be handled without stopping volume servers and
deleting files by hand.
Default action is unmount; --delete also removes the shard files. Targets
resolve against the live topology, with shardId:host:port to disambiguate
co-located or over-replicated shards. Dry-run by default; pass --apply.
2026-07-02 11:58:08 -07:00
Konstantin LebedevGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Konstantin LebedevChris Lu
* feat(k8s): add certificates.dnsNames to inject custom SANs in cert-manager certs
Add certificates.dnsNames configuration option that allows users to
inject custom Subject Alternative Names (SANs) into all cert-manager
Certificate resources. This enables exposing SeaweedFS components
under custom hostnames/CN that aren't covered by the default
wildcard patterns (e.g., '*.filer.default.svc').
The dnsNames list is iterated over in all 6 cert templates
(admin, client, filer, master, volume, worker) and appended to
the spec.x509.subject.names list.
Refs: PR #10035 (original fix-grpc-filer)
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* k8s: quote certificates.dnsNames entries so wildcard SANs render valid YAML
---------
Co-authored-by: MorezMartin <martin.morez@morez.org>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: embedded IAM inline policy honors prefix-scoped resources
getActions stripped the trailing wildcard from a resource like
arn:aws:s3:::bucket/prefix/*, producing a non-wildcard action
(Write:bucket/prefix) that CanDo only ever matched at bucket level, so
PutObject under the prefix was denied. Preserve the object path with its
wildcard (Write:bucket/prefix/*) to match objects under the prefix,
matching the standalone iamapi behavior.
* s3: prune bucket-confined wildcard actions on bucket delete
actionScopedToBucket treated any wildcard as multi-bucket, so a
prefix-scoped action like Write:bucket/prefix/* survived deletion of its
own bucket and could re-grant access if the bucket was recreated. Scope
the wildcard check to the bucket segment only: a wildcard in the object
path stays scoped to its bucket, while one in the bucket segment does
not.
* fix(volume [rust]): compare live compaction_revision instead of stale last_compact_revision
* fix(volume [rust]): compare live compaction_revision instead of stale last_compact_revision - unit tests
* s3: invalidate stale reader cache locations on chunk read failure (#10156)
* s3: invalidate stale reader cache locations on chunk read failure
* filer: share the chunk-read self-heal across reader cache and streaming paths
The reader cache retry added a third copy of the invalidate-relookup-compare-retry
dance already inlined in PrepareStreamContentWithThrottler and duplicated in
retryWithCacheInvalidation. Extract retryFetchWithFreshLocations and route all
three through it, parameterized by the refetch primitive.
* filer: drop redundant completedTimeNew store in reader cache success path
startCaching already stamps completedTimeNew unconditionally before the
fetchErr branch; the second store inside the success branch is dead.
* filer: make NewReaderCache cache invalidator an explicit parameter
The variadic ...CacheInvalidator only ever read the first element, so a caller
could pass two and silently get one. Take a single explicit argument and have
the non-S3 callers pass nil.
* filer: inject reader cache chunk fetch as a struct field
Replace the process-global readerCacheFetchChunkData test seam with a
per-instance fetchChunkDataFn field defaulted in NewReaderCache, matching how
lookupFileIdFn is already wired. Tests set the field on the cache instead of
swapping a shared global.
* filer: log the location count, not full URLs, on self-heal retry
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(shell): honor explicit fs.mergeVolumes from/to direction (#10159)
* fix(shell): honor explicit fs.mergeVolumes from/to direction
mergeVolumes only ever merged a smaller volume into a larger one. When the
user named both -fromVolumeId and -toVolumeId with the source larger than the
target, the planner produced an empty plan and the command printed just
"max volume size: N MB" and moved nothing.
Build the requested pair directly when both ids are given, instead of routing
through the size-descending heuristic. Read-only, empty, and wrong-collection
endpoints are rejected with a clear error rather than a silent no-op.
* fix(shell): allow fs.mergeVolumes into an empty target volume
Merging chunks into an empty volume is valid, e.g. consolidating data into a
freshly created or recently vacuumed volume. Only reject an empty source, which
has nothing to move.
* fix(shell): reject self-map in directed mergeVolumes planner
createMergePlan with from == to returned a {vid: vid} self-merge when called
directly. Guard it in the planner so it is correct independent of the Do
entrypoint.
* fix(volume [rust]): compare compaction_revision in u32, not truncated u16
`req.compaction_revision as u16` truncates any request value above 65535, so a
stale revision of 65537 aliases to a live revision of 1 and the "is compacted"
guard wrongly passes. Widen the volume's revision to u32 and compare there,
matching Go's uint32(v.CompactionRevision) != req.CompactionRevision.
---------
Co-authored-by: adri <adri@digitalunited.net>
Co-authored-by: Aleksey <48918167+MilanFun@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
filer: count reader on cacher waitgroup under the map lock
The read's wg.Add(1) ran after ReadChunkAt released the ReaderCache lock, so a
concurrent destroy() (error eviction, LRU, or UnCache) could start wg.Wait() on
a zero counter and then race the Add - a WaitGroup reuse that trips -race and
can panic. Move the Add under the lock, before the cacher can leave the
downloaders map, so a destroy is always ordered against a counted read.
* s3: add IncompleteBody error code
* operation: tag a truncated source read as ErrTruncatedBody
The chunked uploader wraps both source-read failures and volume-upload failures,
and a mid-write volume server drop also carries io.ErrUnexpectedEOF. Tag only the
source read so callers can tell a truncated input apart from a server-side fault.
* s3: return IncompleteBody for a truncated PUT body
A client abort or reverse-proxy timeout truncates the request body mid-upload.
putToFiler mapped every streaming-upload failure to InternalError (500), which a
reverse proxy relays as a 502. Classify a source-read truncation as IncompleteBody
(400) so the response matches AWS and passes through. All S3 write paths share
putToFiler, so they all benefit.
* volume: drop stale volume-location cache on under-replication
A replicated write looks up the volume's locations and caches them for 10
minutes. When the master briefly reports fewer replicas than the copy count
(e.g. a stale heartbeat drops a just-added volume), that under-replicated
result got cached, so every write failed with "replicating operations is less
than replication copy count" until the entry expired -- long after the master
re-registered the replica.
Invalidate the cached entry when the location count is below the copy count, so
the next write re-queries the master and recovers as soon as it heals.
* volume: mirror the replication copy-count guard in seaweed-volume
do_replicated_request accepted a write even when the master reported fewer
locations than the volume's copy count, silently under-replicating. Reject it,
matching Go's GetWritableRemoteReplications. lookup_volume is uncached, so the
next write recovers as soon as the missing replica re-registers.
* fix(ec): read chunk-manifest chunks stored on EC volumes
Chunk-manifest expansion read every chunk through store.read_volume_needle,
which only resolves a local regular volume. Once a chunk's volume is
EC-encoded, that lookup returns NotFound and the GET fails 500 with
"read chunk ...: not found", so a chunked object over an EC tier is
unreadable even though its parity is intact and reconstructable.
Resolve each chunk to wherever it lives — a local regular volume, a
local EC volume (reconstruct-on-read from the surviving shards), or a
peer via master lookup — matching Go's ChunkedFileReader, which never
assumes chunks are local regular needles.
* fix(ec): validate the chunk cookie on local manifest chunk reads
A chunk fetched from a peer is cookie-checked by that peer's GET handler,
but the local regular and EC reads returned data without comparing the
needle's cookie to the one in the chunk fid. Check it, matching the main
GET paths, so a stale or guessed id can't serve another needle's bytes.
* fix(ec): clamp manifest chunk copy to its declared size
Expansion writes each chunk into result[offset..] by offset, so a chunk
whose bytes exceed its declared size could overwrite the next chunk's
window. Clamp the copy to chunk.size (and reject a negative size) so an
over-long or malformed chunk stays within its own range.
* remote.meta.sync: materialize directory entries, including empty ones
Pull metadata by walking the remote tree one directory level at a time
with a delimiter, so subdirectories, including empty ones, are listed as
their own entries and created locally. The previous flat listing only
returned files, so empty remote directories never appeared locally and
non-empty ones only existed as filer-synthesized parents.
* remote.meta.sync: remove local metadata for entries deleted from remote
After reconciling each directory, drop local entries whose remote source
is gone: files are deleted outright, and a directory removed from the
remote is descended into so its remote-backed children are cleaned while
local-only entries are kept. remote.meta.sync exposes -delete (default
on) and remote.mount.buckets reconciles the same way; a plain
remote.mount stays additive.
* remote.meta.sync: reconcile type swaps and prune emptied directories
- when the remote swaps an entry's type (file <-> directory), drop the
stale local entry and recreate it with the right type; local-only
entries are left alone
- mark synced directories remote-backed and clean a directory removed
from the remote locally, deleting it once it holds no local-only
entries, instead of re-listing the missing remote path
- treat a differing remote size or mtime, not only a newer mtime, as a
change worth pulling
* Respect filerGroup in admin discovery
Admin discovery previously queried master cluster nodes with an empty filer group, so filers registered under a non-default group could not appear in the admin UI. Add an admin filerGroup flag and carry it through cluster-node discovery requests while preserving the empty default behavior.
Constraint: SeaweedFS master ListClusterNodes filters by exact filer_group.
Rejected: Discover all groups implicitly | no existing admin or shell behavior exposes cross-group discovery.
Confidence: high
Scope-risk: narrow
Directive: Keep admin cluster discovery scoped to the configured filerGroup unless an explicit all-groups API is added.
Tested: docker run --rm -v "$PWD:/src" -w /src golang:1.25 go test ./weed/admin/dash -run TestListClusterNodesRequest -count=1
Tested: docker run --rm -v "$PWD:/src" -w /src golang:1.25 go test ./weed/command -run '^$' -count=1
Not-tested: full repository test suite
* mini: pass filer group to admin cluster discovery
miniAdminOptions.filerGroup was never initialized, so startAdminServer
dereferenced a nil *string. Share the filer.filerGroup flag pointer so the
co-located admin queries the same group the filer registers under.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat(filer.sync.verify): reclassify chunk-slice-order ETag diffs as CHUNK_REORDER
filer.ETagChunks concatenates per-chunk MD5s in stored slice order without
normalising by offset, so byte-identical content written by two different paths
(S3 multipart part-completion order on the source vs filer.backup replication
arrival order on the destination) yields different file ETags. filer.sync.verify
reported these as ETAG_MISMATCH even though the files are equal.
Add a second-pass check on every ETAG_MISMATCH: when both sides derive their ETag
from chunks (no attr.Md5) and hold a manifest-free, non-overlapping chunk set
that, once sorted by offset, matches element-wise on (offset, size, ETag),
classify the file as CHUNK_REORDER. Such files are content-equal, so they are
not counted as errors and do not affect the exit code; they are listed only at
higher verbosity (weed -v=1), while the summary always shows their count.
The check stays conservative: a stored attr.Md5 (order-independent content
hash), a differing chunk count, an overlapping/duplicate offset (whose visible
bytes are resolved by timestamp), or a manifest chunk all remain ETAG_MISMATCH.
* filer.sync.verify: decline chunk-reorder fast path on empty per-chunk ETag
An empty or undecodable per-chunk ETag is not a content fingerprint, so
element-wise (offset, size, ETag) equality can't prove the bytes match.
Treating "" == "" as content-equal could reclassify a genuine divergence as
CHUNK_REORDER and drop it from the error count. Decline such chunk sets so they
stay ETAG_MISMATCH.
* filer.sync.verify: emit CHUNK_REORDER in JSON output regardless of -v
The -v=1 gate belongs to the human text report only. Applying it before the
jsonOutput branch dropped the per-file CHUNK_REORDER records from NDJSON while
the summary still counted them, so a machine consumer saw a non-zero count with
no records to reconcile it. Gate the text path only; JSON always emits.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
deps: bump x/image to v0.43.0, repin thrift to CVE-fixed 32-bit-safe commit
x/image v0.41.0 -> v0.43.0 clears CVE-2026-33813/-46601/-46602/-46604.
thrift's only Go-affecting advisory, CVE-2026-41602 (HIGH), is fixed in
v0.23.0, but that release compares an int against untyped math.MaxUint32 in
framed_transport.go and fails to compile on 32-bit (linux/386, arm/v7).
Upstream fixed the range check post-release without tagging it, so replace
now points at that commit: it carries the CVE fix and builds on 32-bit.
Revert to a plain require once thrift tags a release past v0.23.0.
* ec: expose force_deleted_needles_check in ScrubEcVolume RPC and shell
FULL EC scrubs can opt into strict deleted-needle verification via the
-forceDeletedNeedlesCheck shell flag, off by default since it can report
false positives when EC indexes disagree. Rejected for non-FULL modes.
The Rust volume server parses the new field and ignores it: its FULL
scrub verifies shards via RS parity, not per-needle reads.
* volume: require admin auth for ScrubEcVolume
ScrubEcVolume ran unauthenticated while its sibling ScrubVolume, and the
rest of the mutating volume handlers, gate on checkGrpcAdminAuth. Close
the gap so an EC scrub can't be triggered anonymously.
* shell: reject ec.scrub -forceDeletedNeedlesCheck outside full mode
Fail in the client before fanning out to every volume server, instead of
erroring halfway through once the servers reject the request.
* balance: extract the bytes-aware density metric to weed/topology/balancer
The shell's volume.balance ranks servers by a bytes-aware density (used volume
equivalents over free capacity). Move that math into the shared balancer package
(VolumeDensity / DensityRatio / DensityNextRatio) so the maintenance worker can
adopt the same metric next. Shell behavior is unchanged.
* balance: rank a server with no free capacity as the fullest
DensityRatio/DensityNextRatio divided by capacity, so a server past its slot
limit (negative capacity) returned a negative ratio and sorted as the emptiest
under ascending consumers — the opposite of reality. Treat any non-positive
capacity (full, or overfull mid-run after receiving volumes) as the fullest
(+Inf) so it is a move source, never a target. Covered by negative-capacity and
ordering tests.
* kms: cache decrypted data keys so cache_enabled/cache_ttl take effect
cache_enabled/cache_ttl were parsed into KMSConfig but never consulted,
so every SSE-KMS read repeated a KMS Decrypt round-trip. Add a
CachedKMSProvider decorator keyed on (ciphertext, encryption context)
and wrap providers with it wherever one is created. Decrypt is
deterministic for that pair; GenerateDataKey/DescribeKey/GetKeyID pass
through untouched.
Cache hits and stores return private copies so the read path's
ClearSensitiveData can't wipe the cached key. TTL and max_cache_size
bound the cache.
* kms: zero superseded data keys and pool cache-key hash states
Wipe the old plaintext when a cache entry is overwritten so a key
material buffer left behind by two readers racing on the same miss does
not linger in memory. Reuse sha256 states via a sync.Pool so the read
hot path stops allocating a fresh hash on every Decrypt.
* kms: drop cache writes after Close
An in-flight Decrypt miss can call set after Close scrubbed the map,
repopulating it with a data key that would then never be cleared. Guard
set with a closed flag so post-Close stores wipe their copy and no-op.
* admin: never close the worker outgoing channel while senders are live
conn.outgoing has multiple concurrent senders (heartbeat, task assignment,
log request, registration handlers). Closing it on connection teardown raced
a sender and paniced with "send on closed channel" — reliably reproduced when
a laptop goes idle: heartbeats stall past the 2-minute stale cutoff, the
cleanup routine closes the channel, and the resumed worker's heartbeat is
received and handled at the same moment.
The connection context is already the sole teardown signal, so stop closing
the channel entirely. handleOutgoingMessages exits on conn.ctx.Done(), and the
buffered channel is GC'd once the connection drops. Route sends through a
sendToWorker helper that also selects on conn.ctx.Done() so they bail on
teardown instead of blocking for the full timeout.
* admin: bail on ctx.Done() while waiting for a worker log response
* refactor: centralize genUploadUrl in UploadOption
Replace inline genFileUrlFn closures with operation.GenUploadUrl field:
- Add GenUploadUrl func(host, fileId) string to UploadOption struct
- Add GenUploadUrlProxy(filerAddress string) utility function
- Remove genFileUrlFn parameter from UploadWithRetry signature
- Update all callers: mount, gateway, mq, filer_copy, filer_sync
This matches the weed mount -filerProxy pattern exactly,
factorizing the URL generation logic across all consumers.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* docker release: run all platform jobs in one wave, cache rocksdb compile
Drop max-parallel so the 13 per-platform builds run together instead of two
waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late).
Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is
sha-independent, so it caches across releases and stops being the ~16-min
long-pole that gates the merge fan-in. go-build variants stay mode=min.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* refactor: centralize genUploadUrl in UploadOption
Replace inline genFileUrlFn closures with operation.GenUploadUrl field:
- Add GenUploadUrl func(host, fileId) string to UploadOption struct
- Add GenUploadUrlProxy(filerAddress string) utility function
- Remove genFileUrlFn parameter from UploadWithRetry signature
- Update all callers: mount, gateway, mq, filer_copy, filer_sync
This matches the weed mount -filerProxy pattern exactly,
factorizing the URL generation logic across all consumers.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* Remove accidental ROCmFPX submodule reference
* gofmt chunk upload option block
* Preserve broker cipher and re-read proxy filer per upload attempt
Chunk uploads must keep the configured Cipher, and both the mount and broker current filer can change on failover, so build the proxy upload URL inside the closure instead of capturing the address once.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
save_bitrot_sidecar writes payload.len() into the header as a u32; guard against
a payload > 1 GiB (which would silently truncate the length field), mirroring
Go's SaveBitrotSidecar maxBitrotPayloadSize check. The check uses encoded_len()
before serializing, so an oversized manifest never allocates a large buffer.
Never triggers for a real sidecar (a few KB).
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* proto: add EC bitrot checksum messages + CHECKSUM scrub mode
Mirror weed/pb/volume_server.proto byte-for-byte (field numbers + types) so the
.ecsum sidecar payload is wire-identical across the Go and Rust binaries:
EcBitrotProtection / EcShardChecksums / ChecksumAlgorithm, VolumeScrubMode.CHECKSUM=4,
and VolumeEcShardsCopyRequest.copy_ecsum_file. No code uses them yet — the .ecsum
format, producer, mount-load, copy, and scrub land in following commits.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): port the .ecsum bitrot checksum module
ec_bitrot.rs mirrors weed/storage/erasure_coding/ec_bitrot.go: the .ecsum sidecar
format (14-byte big-endian ECSU header + CRC32C over a prost-serialized
EcBitrotProtection payload), the per-shard per-block CRC32C producer
(ShardChecksumBuilder), save/load with payload self-integrity, manifest
validation, status resolution, and verify_shard_file_blocks for the CHECKSUM
scrub. A byte-exact test pins the serialized bytes against the Go reference's
identical constant so a format drift in either binary fails loudly.
Producer wiring (encode/vacuum), mount-load, copy, and the mode-4 dispatch land
in following commits.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* test(ec): pin .ecsum sidecar bytes for cross-binary interop
Deterministic EcBitrotProtection -> exact on-disk bytes, asserted against a
canonical constant on BOTH sides (this test and ec_bitrot.rs), so a format drift
in either binary fails its own suite rather than silently desyncing a Go-written
.ecsum from a Rust-written one.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): write the .ecsum bitrot sidecar during EC encode
write_ec_files now feeds each shard's bytes through a per-shard
ShardChecksumBuilder as it writes them, then persists the generation-0 sidecar
(<base>.ecsum) alongside the shards — mirroring weed's WriteEcFiles +
SaveBitrotSidecar. Best-effort: a failed sidecar write leaves the generation
unprotected rather than failing the encode. A test confirms the produced sidecar
validates and its per-block CRCs match every on-disk shard.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): load the .ecsum at mount + EcVolume::checksum_scrub
EcVolume now loads and validates its generation-0 .ecsum sidecar at mount,
caching the parsed protection + BitrotStatus (Off/On/Invalid), and exposes
bitrot_protection() mirroring Go's EcVolume.BitrotProtection(). checksum_scrub()
verifies every locally-held shard's raw bytes against the sidecar block CRCs —
the only path that exercises cold parity shards — reporting mismatched shards
without mutating anything; a wholesale mismatch beyond parity is flagged as a
suspect sidecar rather than mass shard corruption. Mirrors Go's ChecksumScrub.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): dispatch EC CHECKSUM (mode 4) to checksum_scrub
Accept VolumeScrubMode.CHECKSUM=4 and route it to EcVolume::checksum_scrub,
accumulating blocks scanned + mismatched shards into the scrub response, plus the
CHECKSUM scrub-mode metric label. Read-only bitrot verification over local shards.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): copy the .ecsum sidecar during VolumeEcShardsCopy
Honor copy_ecsum_file: when set, copy the generation-0 .ecsum alongside the
shards so protection travels with them, mirroring Go's non-2PC copy path.
Tolerant of a missing source (empty stream) — the 0-byte file is dropped so
mount sees no sidecar (protection off) rather than a truncated/invalid one.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): remove the .ecsum sidecars when destroying an EC volume
remove_ec_volume_files now clears <base>.ecsum (and any versioned .ecsum.v<N>)
from the data and idx dirs, so a vid reuse can't load a stale sidecar. Mirrors
Go's removeBitrotSidecars.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* style(ec): align bitrot comments and test setup for merge-cleanliness
Match the shared bitrot code (write_ec_files, encode_one_batch, checksum_scrub,
the encode sidecar test) to the canonical wording/layout so the volume-server
Rust port stays line-aligned across trees, keeping periodic merges conflict-free.
No behavior change.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
The disk-fullness gate only rejected destinations already at/above the mark, so a
server just under it could take a large volume and overshoot. Project the selected
volume's bytes onto the candidate: if the move would cross the mark, drop that
destination for the rest of the cycle and re-pick instead of overshooting. Also
note the per-location capacity-summing assumption on the Rust heartbeat side, to
match the Go store.go comment.
The replica-placement rule (data-center/rack/same-node limits plus host
anti-affinity) existed three times: the shell's satisfyReplicaPlacement/isGoodMove
used by volume.balance, fix.replication, and tier.move, and a line-for-line port
in the maintenance balance worker. Move the canonical logic into
weed/topology/balancer on a shared Location type; the shell and worker keep thin
adapters that convert their own location representation and call it. Behavior is
unchanged (the shared IsGoodMove keeps the shell's reject-move-to-self guard, and
all four replica test suites pass).
EC placement scored destinations purely by free EC shard slots (derived from
maxVolumeCount) and shard counts, blind to real disk fullness — the same defect
as volume balancing. A disk that is physically full but still shows free EC slots
kept being chosen, and EC shard bytes are captured by statfs free space yet not
by any slot accounting, so the slot math is exactly the metric that can't see EC
fullness.
Treat a disk at/above 90% physical usage as having zero free EC slots at
snapshot-build time, so every existing freeSlots>0 placement predicate excludes
it. Applied in all three snapshot builders (shell countFreeShardSlots, the shared
ecbalancer FromActiveTopology, and the worker ec_balance buildBalancerTopology)
via the shared balancer.DiskTooFullAfter gate. Servers not reporting disk bytes
fall back to slot-only behavior. ec.rebuild recovery is left ungated so shard
recovery can still complete onto fuller disks.
* shell: add volume.balance -byDiskUsage to balance by actual data
The default balancer ranks servers by slot density, dividing used volumes by
MaxVolumeCount. When MaxVolumeCount is configured higher than the disk can hold,
a physically near-full server looks nearly empty and gets picked as the move
target, so balancing drains less-full servers onto an already-full one.
-byDiskUsage ranks servers by the actual data they hold (sum of volume sizes)
instead, so the fullest-by-data server is treated as full and balancing drains
it. It assumes comparable disk sizes per disk type and still respects each
server's free volume slots. Default behavior is unchanged.
* plumb physical disk usage into topology, gate volume.balance on it
Volume servers now report each disk's filesystem total/free bytes in the
heartbeat, and the master stores them in DiskInfo. volume.balance uses them to
skip any move target whose disk is already near full (-maxDiskUsagePercent,
default 90), so an over-configured maxVolumeCount can no longer make a
physically full server look empty and get drained onto. The gate judges each
server against its own disk, so heterogeneous disk sizes are fine; servers that
do not report bytes fall back to slot-only behavior.
Rust seaweed-volume mirrors the heartbeat reporting.
* admin: report real physical disk capacity when volume servers provide it
The dashboard estimated server capacity as maxVolumeCount * volumeSizeLimit,
which overstates it when maxVolumeCount is set higher than the disk holds.
Prefer the filesystem capacity now reported per disk, falling back to the
estimate for servers that do not report it.
* worker: gate automatic balance on physical disk fullness too
The maintenance balance worker selects the least slot-utilized server as the
move destination, so an over-configured maxVolumeCount makes a physically full
server look empty and get drained onto — the same defect as the shell command.
Now that DiskInfo carries real disk bytes, skip any destination whose disk is
at/above 90% used (per server, against its own disk); a full server can still be
a source. When every candidate destination is full, create no tasks. Servers
that do not report disk bytes are not gated.
* balance: share the physical-disk-fullness gate between shell and worker
The shell volume.balance command and the maintenance balance worker each grew
their own copy of the disk-fullness gate (targetDiskTooFull / destinationDiskTooFull)
and a maxDiskUsagePercent=90 constant. Pull both into weed/topology/balancer
(DiskTooFullAfter + DefaultMaxDiskUsagePercent) so the policy has one home and the
two balancers can't drift.
* balance: harden the physical-disk gate
Guard against a nil DiskInfo in the byte/slot lookups. Let a zero disk-capacity
report clear previously stored bytes (0 means "not reported" for bytes, unlike
maxVolumeCount), so a server that stops reporting falls back to slot-only instead
of trusting stale capacity. In the worker, charge each planned move's bytes to
its destination within a detection cycle so the gate sees a target fill up rather
than only its heartbeat-time free space. Note the per-location capacity summing
assumes one location per filesystem (the used ratio the gate relies on stays
correct regardless; absolute capacity can over-report).
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* proto: per-disk type/capacity in DiskTag, DiskInfo.physical_disks
DiskTag gains type + max_volume_count so the heartbeat can describe every
physical disk, including ones holding no volumes or EC shards. DiskInfo
gains physical_disks so the master can hand the full per-type disk set to
per-physical-disk consumers.
* feat(volume): report each physical disk's type and capacity
CollectHeartbeat fills DiskTag.type and the per-disk effective max for
every location, so the master can account for disks that hold no volumes
or EC shards yet. Rust heartbeat mirrors it.
* feat(master): surface empty disks in the per-physical-disk view
The master records each disk's type and max from DiskTags and lists them
on DiskInfo.physical_disks per type, including disks with no volumes or
EC shards. SplitByPhysicalDisk enumerates that full set and gives each
disk its exact max, so cluster.status, volume.list and the admin
topology count and can target empty disks. Without physical_disks the
even-split fallback is unchanged.
* fix(master): clamp per-disk free at zero for over-allocated disks
In the exact-max path FreeVolumeCount could go negative when a disk holds
more volumes than its max; a negative would reduce the node's summed free
and block placement on healthy disks. Clamp at 0.
* fix(master): rebuild disk tags fresh each heartbeat
DiskTags is the full authoritative per-disk list every heartbeat, so
rebuild dn.diskTags from scratch like dn.diskBackends; merging left stale
entries for removed disks.
* fix(master): keep zero-capacity disks in physical_disks
A disk reporting max 0 (an unavailable disk) is a valid physical disk,
not a signal to drop it. List every disk of the type, but only emit
physical_disks when the node reports real per-disk capacity, so an older
server sending all zeros still falls back to the aggregate split.
* test(volume): cover disk-space-low per-disk max in heartbeat
Assert DiskTag.max_volume_count follows the used-slots override when a
location is low on space, matching the per-type max_volume_counts.
* chore: trim comments on the empty-disk change
Drop narration; keep only the non-obvious why (disk-0 sentinel, exact-max
free clamp, EC slots not subtracted, all-zeros fallback).
* refactor(master): merge per-disk tags and capacity into one map
diskTags and diskBackends were parallel maps keyed by the same DiskId and
filled together from DiskTags. Fold them into one diskMetas map of
{tags, type, max}.
* refactor(proto): per-disk max as a map keyed by disk id
physical_disks was a repeated {disk_id, max_volume_count} whose fields
duplicated DiskInfo's own disk_id/max_volume_count. A map<uint32,int64>
keyed by disk id expresses "max per disk" directly, drops the extra
PhysicalDiskInfo message, and the consumer reads it as the disk set.
* docs(proto): note DiskInfo.disk_id's two meanings
Identity on a per-physical-disk DiskInfo (from SplitByPhysicalDisk),
representative fallback on the type-keyed aggregate.
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* fix(shell): honor explicit fs.mergeVolumes from/to direction
mergeVolumes only ever merged a smaller volume into a larger one. When the
user named both -fromVolumeId and -toVolumeId with the source larger than the
target, the planner produced an empty plan and the command printed just
"max volume size: N MB" and moved nothing.
Build the requested pair directly when both ids are given, instead of routing
through the size-descending heuristic. Read-only, empty, and wrong-collection
endpoints are rejected with a clear error rather than a silent no-op.
* fix(shell): allow fs.mergeVolumes into an empty target volume
Merging chunks into an empty volume is valid, e.g. consolidating data into a
freshly created or recently vacuumed volume. Only reject an empty source, which
has nothing to move.
* fix(shell): reject self-map in directed mergeVolumes planner
createMergePlan with from == to returned a {vid: vid} self-merge when called
directly. Guard it in the planner so it is correct independent of the Do
entrypoint.
* s3: invalidate stale reader cache locations on chunk read failure
* filer: share the chunk-read self-heal across reader cache and streaming paths
The reader cache retry added a third copy of the invalidate-relookup-compare-retry
dance already inlined in PrepareStreamContentWithThrottler and duplicated in
retryWithCacheInvalidation. Extract retryFetchWithFreshLocations and route all
three through it, parameterized by the refetch primitive.
* filer: drop redundant completedTimeNew store in reader cache success path
startCaching already stamps completedTimeNew unconditionally before the
fetchErr branch; the second store inside the success branch is dead.
* filer: make NewReaderCache cache invalidator an explicit parameter
The variadic ...CacheInvalidator only ever read the first element, so a caller
could pass two and silently get one. Take a single explicit argument and have
the non-S3 callers pass nil.
* filer: inject reader cache chunk fetch as a struct field
Replace the process-global readerCacheFetchChunkData test seam with a
per-instance fetchChunkDataFn field defaulted in NewReaderCache, matching how
lookupFileIdFn is already wired. Tests set the field on the cache instead of
swapping a shared global.
* filer: log the location count, not full URLs, on self-heal retry
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Concurrent bucket deletion across multiple filer replicas races on the
per-bucket DROP TABLE. The first replica drops the table; the rest hit
an undefined-table error (postgres 42P01, mysql 1051) which propagates
out of DeleteFolderChildren and panics the filer. On restart the same
pending DROP re-runs and the filer crash-loops.
Make the drop idempotent. Same defect class in all SQL backends, so fix
postgres, postgres2, mysql, mysql2, and sqlite together.
* fix(ec): correct EC FULL scrub for deleted needles + shard-location cache
Addresses review findings on the EC FULL distributed scrub:
- Remote EC reads now thread Go's (bytes, is_deleted) contract. A runtime EC
delete keeps the .ecx size positive (the delete lives in .ecj/memory), so the
raw-index walk verifies the needle, and its header interval is usually remote;
the peer answers is_deleted with no payload. The scrub zero-fills that interval
(so the needle reaches read_bytes -> SizeMismatch{0} -> the delete-state
suppression), the serving direct read short-circuits to not-found, and
reconstruction EXCLUDES the shard instead of feeding zeros into Reed-Solomon.
- The walk skips size.is_deleted() (not just is_tombstone), so a -originalSize
.ecx entry (pre-encode delete) can't yield empty intervals or panic parse_header.
- Restore Go's < data_shards completeness guard (per-volume, custom-ratio aware)
and per-shard merge in the location cache instead of clobber-with-partial.
- Abort the scrub with an error on mid-scan unmount instead of a false-CLEAN.
- Hoist the refreshed location map once instead of cloning it per needle.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): keep RS parity check in EC FULL until CHECKSUM lands
The per-needle FULL walk only reads live data-shard intervals, so it can't catch
bitrot in a parity shard or an unwalked cold region. Run verify_ec_shards
alongside the walk, gated on all-shards-local (single-node EC), via spawn_blocking.
A deliberate temporary divergence from Go FULL; moves to mode 4 (CHECKSUM) once
the .ecsum subsystem lands.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): add scrub_ec_volume_distributed (FULL EC scrub, local+remote)
Ports Go's Store.ScrubEcVolume: walk the raw .ecx, verify every needle across
local AND remote shards without decoding (report faults, don't heal), with the
#10130 deleted-needle size-mismatch suppression gated on a force flag. Reuses
the read path's lock-drop + no-reconstruct read_remote_ec_shard_interval so no
!Send store guard is held across an .await.
Walks the unmasked index (scrub_snapshot_under_lock locates from the raw
(offset, size), not locate_needle) so logically-deleted-but-present needles are
still byte-verified, matching Go. Refreshes shard locations once up front and
hard-fails on a master-lookup error rather than retrying per needle.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): dispatch EC FULL (mode 2) to the distributed needle walk
FULL ran a local-only Reed-Solomon parity check; route it to the per-needle
local+remote walk instead, mirroring Go. The handler collects vids under a brief
lock then releases it: FULL self-locks per needle (it awaits remote reads),
INDEX/LOCAL re-acquire a brief lock. verify_ec_shards is retained but no longer
wired to a mode.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
The EC volumes, EC shards, and collection details pages each rendered a
repair (wrench) button for incomplete EC volumes. Its handler POSTed to a
/repair endpoint that the admin server never registers, so every click
returned "404 page not found" (the collection details page only had a
placeholder handler).
Remove the buttons and their JavaScript handlers, and regenerate the
templ output. Manual EC shard recovery remains available from weed shell
via ec.rebuild.
EC volumes do not propagate deletions to all shard indexes, so it is possible
to run scrubbing on a volume where a deleted needle is still present in the
index, or a needle deleted from the index is still present on the volume.
On either scenario, scrubbing will fail due to size mismatch errors.
This PR reworks the scrubbing logic so needle size mismatches are
ignored in such scenarios.
Scrubbing can still be forced to check deleted needles (f.ex. to discover
index inconsistencies); this option will be exposed in RPCs and `weed shell`
on a follow-up PR.
* fix(scrub): don't flag offset-0 logical tombstones in volume scrub
A remote-tier delete records a tombstone at .idx offset 0 with no physical .dat
bytes. Full scrub double-flagged a healthy remote-tiered volume with deletes:
scrubVolumeData counted the tombstone's GetActualSize(-1)=32 toward totalRead
(want > physical .dat), and CheckIndexFile treated it as occupying [0,31] and
flagged the first live needle as overlapping. Skip offset-0 logical tombstones
from both the size reconcile and the overlap check; they are still counted for
the index-size check. Local deletes (offset != 0) are unaffected.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(scrub): mirror offset-0 logical tombstone handling into Rust
Same fix as the Go volume_checking.go + idx/check.go change: Volume::scrub skips
offset-0 logical tombstones from total_read, and check_index_file excludes them
from the overlap check (still counted for the index-size check).
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(ec): suppress deleted-needle size mismatch in EC LOCAL scrub
EcVolume.ScrubLocal reassembles each fully-local needle and ReadBytes-checks
it, but appended every error unconditionally. A needle the .ecx still reports
live while its reassembled on-disk header carries size 0 (delete state
disagrees between index and header) is not corruption — the LOCAL twin of the
#10130 fix for the FULL path. Suppress the ErrorSizeMismatch in that case;
genuine (non-zero) size mismatches and CRC/tail errors are still reported.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(ec): mirror EC LOCAL scrub deleted-needle suppression into Rust
Same suppression as the Go EcVolume.ScrubLocal change: a NeedleError::SizeMismatch
whose on-disk header size is 0 against a live index entry is a delete-state
disagreement, not corruption.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): extract locate_ec_shard_needle_interval
Mirrors Go's EcVolume.LocateEcShardNeedleInterval; reused by locate_needle
and the upcoming local scrub walk.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): add EcVolumeShard::to_ec_shard_info
Mirrors Go's ToEcShardInfo.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): add EcVolume::scrub_local
Walk the .ecx and verify each needle against the locally-held shards,
reading interval-by-interval (reusing one chunk buffer); CRC-check only
fully-local needles, report short/unreadable local shards, and abort the
scan on a structural size mismatch. Mirrors Go's EcVolume.ScrubLocal.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): dispatch EC LOCAL (mode 3) to scrub_local
Splits the mode 2|3 arm: FULL (2) keeps the Reed-Solomon parity check;
LOCAL (3) now runs the per-needle local-shard walk.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* refactor(scrub): extract open_index_for_scrub shared by scrub_index
Mirrors Go's openIndex, shared by ScrubIndex and the upcoming Scrub rewrite.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(scrub): walk the on-disk .idx in Volume::scrub
scrub walked the deduped in-memory map, so total_read undercounted the
physical .dat on any volume with overwrites or deletes and the size
reconcile falsely flagged healthy volumes broken. Walk every .idx row
instead (matching Go's scrubVolumeData): count all rows, CRC-verify live
needles, skip deleted, and reconcile against the .dat. Holds one data-file
read lock and reads via the unlocked path, like Go's Scrub.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(idx): add check_index_file mirroring Go idx.CheckIndexFile
Index-only structural check: walk the on-disk index, sort by (offset, size),
flag overlapping needles, and verify the file is a whole number of entries.
No data-file reads.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* refactor(ec): use idx::check_index_file in EcVolume::scrub_index
Drops the inline walk/sort/overlap copy. Walks a private fd so the structural
scan never moves the shared ecx_file cursor (read positionally elsewhere).
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(scrub): make Volume::scrub_index an index-only check on the on-disk .idx
INDEX mode walked the deduped in-memory map and read .dat headers — more
than the cheap-INDEX contract allows, yet missing Go's overlap and
size-multiple structural checks. Route it through idx::check_index_file so
it matches Go's Volume.ScrubIndex and the INDEX<LOCAL<FULL cost tiering holds.
Ports openIndex's zero-size-index guard (a populated .dat with an empty .idx
is corruption) and takes the data-file read lock for a consistent snapshot.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(ec): cap EcShardConfig at MAX_SHARD_COUNT, not TOTAL_SHARDS_COUNT
read_ec_shard_config rejected any .vif ratio summing past 14 shards and
silently fell back to 10/4, so wider EC volumes ran against the wrong
shard set. Match Go's MaxShardCount(32) bound.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* docs(ec): correct stale 0..14 shard-count comments
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
Master /dir/lookup JSON omits publicUrl when empty (Go json omitempty).
The Rust volume server required the field, so serde failed with "lookup
parse failed: error decoding response body" and cross-DC replicated writes
failed.
Default publicUrl to empty, fall back to url for peer filtering, and
normalize addresses with to_http_address before excluding the local peer
(so host:port.grpcPort forms do not match self incorrectly).
* test(seaweed-volume): cover type=replicate fan-out writes
A holder must accept a replicated copy and store it locally without
re-replicating. Covers raw and multipart bodies, and a multi-copy
volume where re-replication would otherwise reach the master.
* test(seaweed-volume): use port 0 for the dead-master address
Connecting to port 0 is refused at the socket layer immediately, so the
plain-write fan-out path fails fast instead of risking a connect-timeout
hang where port 1 is filtered.
* feat: add collection pattern to delete empty volumes
Co-authored-by: Codex <noreply@openai.com>
* shell: match collection pattern with wildcard matcher
Use wildcard.MatchesWildcard in the shared collection-pattern helper,
matching command_volume_fix_replication's matchCollectionPattern. The
flag only advertises '*' and '?', which is exactly what the matcher
supports.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: support AWS object form for bucket policy Principal, add NotPrincipal
Bucket policy statements only accepted a bare string or array of strings for
the Principal element, so the AWS-documented object form was rejected:
"Principal": { "AWS": "arn:aws:iam::123456789012:root" }
"Principal": { "AWS": ["arn:...", "999999999999"] }
Add a PolicyPrincipal type that parses the bare string, the bare array
(retained for backward compatibility), and the object form keyed by AWS,
Service, Federated or CanonicalUser (each value a string or array). All keyed
values are flattened for principal matching, and the original JSON is preserved
so PutBucketPolicy/GetBucketPolicy returns the exact shape submitted - keeping
infrastructure-as-code tools (Terraform, Ansible) idempotent.
Also add NotPrincipal support (a statement applies to every principal except the
ones named), compiled and evaluated in both policy evaluators, and reject
statements that specify both Principal and NotPrincipal.
* s3: address review - validate principal object form, honor dynamic NotPrincipal
- Reject unsupported Principal object keys (only AWS/Service/Federated/
CanonicalUser) and empty values, so a form like {"AWS":[]} no longer compiles
to zero matchers and silently relies on the match-all fallback.
- Detect both Principal and NotPrincipal by field presence, not by flattened
length, so a present-but-empty field is still rejected.
- Honor dynamic (policy-variable) NotPrincipal/Principal patterns in the
compiled evaluator; previously a NotPrincipal made only of variables was
treated as absent and its exclusion bypassed.
- Add regression tests for the object-form validation and dynamic NotPrincipal.
* Review comment removed unnecessary success and failure count
* fix: use Gather.Gather() with seeded counter for EC rebuild registration test
- Restore Gather.Gather() to verify MustRegister calls as requested in review
- Seed VolumeServerECRebuildCounter before gathering because CounterVec
only appears after at least one label value is observed
- Use correct fully-qualified metric names (SeaweedFS_volumeServer_*)
* fix: remove preflight checkEcVolumeStatus failure from ec_rebuild_total counter
ec_rebuild_total should only reflect actual rebuild execution failures
(from RebuildEcFiles / RebuildEcxFile), not scan/precheck failures in
the volume status loop. The error is still returned to the caller;
only the misleading counter increment was removed.
* Review comment removed unnecessary observe
* label EC rebuild duration histogram by result
Without a result label, fast failures pull down the success-latency
quantiles shown on the EC Rebuild Duration panel. Make the histogram a
HistogramVec keyed by result, record success/failure through one
recordEcRebuild helper, and split the Grafana quantiles by (le, result).
* reset EC rebuild metric vecs in registration test
The HistogramVec needs a child before Gather emits it, so the test must
observe once; reset both vecs in cleanup so that sample doesn't leak into
other tests.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
An empty or truncated tasks/*.pb file unmarshals into a TaskStateFile
with a nil Task, and protobufToMaintenanceTask dereferenced it
immediately, panicking the whole admin process on startup. Guard the
nil case so the loader logs a warning and skips the bad file.
Under a herd of concurrent assigns with no writable volume, Assign spun
PickForWrite for the full 10s timeout, pinning a goroutine per request and
starving the master of the cycles it needs to process growth and answer
heartbeats. When growth is the relevant remedy and already in flight, stop
spinning: if free space exists, shed with a fast retryable error so clients
back off and retry once growth lands; if the cluster is out of space, fail fast
with the real out-of-space error instead of masking it as retryable.
The gRPC shed uses ResourceExhausted, not Unavailable: operation.Assign retries
it, but the client connection layer doesn't treat it as a dead channel, so a
per-request shed across a herd doesn't tear down the shared master connection
and cancel every other in-flight assign. The HTTP dirAssignHandler sheds with
503 + Retry-After.
* volume server: route VolumeMarkReadonly to raft leader
After a master raft election, volume servers may still heartbeat a follower
while admin paths such as weed shell volume.mark call notifyMasterVolumeReadonly
via vs.GetMaster(). Followers reject VolumeMarkReadonly with NotLeader, which
breaks tiering and other mark-readonly workflows until the heartbeat loop
reconnects.
Resolve the leader through GetMasterConfiguration on configured -master peers
(same Leader field filer/master clients already use) before calling
VolumeMarkReadonly. When the leader differs from the heartbeat peer, update
currentMaster so the heartbeat loop converges faster.
Adds operation.LookupRaftLeaderMaster with unit tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: address review feedback on volume.mark raft leader routing
Do not update currentMaster during leader lookup — heartbeat owns that
field and uses stream GetLeader() to reconnect. Try the heartbeat peer
first and only resolve the raft leader after a NotLeader rejection.
Add ctx.Err() early exit and quieter logging for context cancellation.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(operation): thread the lookup timeout ctx into connection invalidation
The 5s timeout drove only the RPC; WithMasterServerClient saw the
unbounded outer ctx, so a self-inflicted timeout (slow GetMasterConfiguration
during an election) was treated as a stale channel and tore down the shared
master connection. Pass the timeout ctx into the helper so its own expiry
leaves ctx.Err() set and spares the connection.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(filer.sync.verify): sort listings client-side before merge
The merge walks both filers' directory listings in lockstep and needs
them in the same byte order. A filer before 4.32 with a locale SQL
collation lists case-insensitively while a 4.32+ peer lists byte-ordered,
so comparing two such clusters returns the same names in a different
order and the merge desyncs into spurious MISSING / ONLY_IN_B.
Buffer and sort each directory client-side so both sides agree on order
regardless of filer version or store backend. Trades the streaming
source's O(buffer) memory for O(directory) per side, fine for a one-shot
verify CLI; both sides still load concurrently.
Claude-Session: https://claude.ai/code/session_01BKsBdKYFNCEjeHLjJfumPF
* fix(filer.sync.verify): surface listing errors before merging
A listing that fails mid-stream leaves a partial, unsorted buffer. Now
that both sides are fully buffered anyway, check each side's error right
after the loads finish and before the merge, so partial entries can't
emit spurious MISSING / ONLY_IN_B before the error aborts the run.
Claude-Session: https://claude.ai/code/session_01BKsBdKYFNCEjeHLjJfumPF
* fix(shell): correct volume.list -writable filter unit and comparison
* fix(shell): correct volume.list -writable filter unit and comparison
* chore(shell): fix typo in EC shard helper param names
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers
* fix: apply collectionPattern during detection in volume.fix.replication
* use existing wildcard.MatchesWildcard for collection matching
It returns a plain bool, so drop the up-front filepath.Match validation
and the path/filepath import that only existed to handle its error.
* trim verbose comments to terse one-liners
* drop redundant per-path collection guards
Detection already filters by replicas[0].info.Collection. The repair guard
re-checked pickOneReplicaToCopyFrom's collection (a different replica), so a
mixed-collection volume could pass detection yet be skipped in repair without
decrementing the counter, spinning the -apply loop. deleteOneVolume keeps its
collectionIsMismatch safety.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
mount: move directory cache state to a side map to shrink InodeEntry
The mount keeps an InodeEntry alive for every inode the kernel references.
On a mount that is almost entirely regular files, each entry carried the full
directory readdir-cache bookkeeping (four time.Time fields plus counters),
bloating it to 152 bytes whether or not the inode was a directory.
Move that state into a dirState held in a side map keyed by inode, and drop the
isDirectory bool: an inode is a directory iff it has a dirState. InodeEntry is
now just paths + nlookup at 32 bytes, landing in a smaller Go allocator size
class; on a mount with tens of millions of cached file inodes that is several GB
less resident heap. As a side effect the readdir-cache scan helpers iterate only
directories instead of every inode.
* fix(volume): fsync .vif and downloaded tier .dat (Rust)
save_volume_info wrote the .vif with a plain write and no fsync, and the
tier download never synced the .dat it wrote. Either could be lost on a
crash before the tier-down path acts on them. fsync both, matching the Go
volume server's util.WriteFile and DownloadFile.
* fix(volume): swap to local before deleting remote on tier-down (Rust)
The tier-down path deleted the shared remote object before trimming the
.vif, so a crash in between left the volume's .vif pointing at a deleted
object. It also dropped the remote backend only on the delete path and
never opened the downloaded local .dat, so reads broke until reload and a
keep-remote download kept serving from the slow remote object.
Trim the .vif and swap to the local .dat on both paths, bracketed by
directory fsyncs, before removing the remote object; gate only the object
removal on keep_remote_dat_file. Matches the Go volume server's crash-safe
ordering.
After VolumeTierMoveDatToRemote uploaded the .dat, the volume closed its
local backend but never opened the remote one, leaving both dat_file and
remote_dat_file empty. The needle read path has no lazy reopen, so reads
returned "dat file not open" until the volume reloaded.
Switch to the remote backend right after saving the .vif, the same as the
Go volume server's LoadRemoteFile, so the volume keeps serving from remote
storage immediately after tiering.
* ci: add per-process memory sampler for perf jobs
Samples VmRSS once a second into a CSV and records peak VmHWM per process
on stop. Linux only; reads /proc/<pid>/status.
* ci: run perf benchmarks on the Rust volume server and report memory
Matrix the throughput and S3 jobs over go/rust volume servers, using a
standalone master (plus filer for S3) and swapping only the volume binary
so the two are directly comparable. Sample peak RSS in every job and surface
it per impl in the run summary.
* ci: harden mem sampler arg handling and peak fallback
Guard against missing args under set -u, and fall back to the max RSS
sampled when a process exits before VmHWM can be read.
* ec: recover EC shards whose .ecx index lives only on a peer server
A volume server that boots with EC shard files on disk but no .ecx index
on any local disk cannot mount the shards, so the master never learns
about them. ec.rebuild works off master-registered shards, so it sees the
volume as short and gives up even though the shard data is intact.
Add an operator-triggered recovery: VolumeEcShardsMount gains a
recover_missing_index flag that makes the volume server fetch the missing
.ecx (plus .ecj/.vif) from a peer holding it and mount the on-disk shards.
ec.rebuild runs this across the cluster before planning, so orphaned
shards register and the rebuild sees the true shard set.
.ecx is an immutable encode-time index, identical on every holder. .ecj
is a per-holder deletion journal that differs across holders, so the
recovered node adopts the source peer's deletion view, like a balanced or
rebuilt shard does.
* ec: mirror missing-index recovery into the Rust volume server
Port the #10104 recovery to seaweed-volume so the Rust volume server
self-heals the same layout: EC shards on disk with the .ecx index only on
a peer. Adds collect_ec_volumes_missing_index / mount_recovered_ec_shards
to the store, recover_missing_ec_indexes (master LookupEcVolume + peer
CopyFile fetch + mount) to the server, and the recover_missing_index flag
on VolumeEcShardsMount.
.ecx is the immutable encode-time index, identical on every holder. .ecj
is a per-holder deletion journal, so the recovered node adopts the source
peer's deletion view, matching the Go path.
* fix(volume): stream copy_file from disk instead of buffering whole file
copy_file pushed every 2MB chunk into a Vec and only then returned tokio_stream::iter(results), so serving a near-limit volume as a copy source (e.g. during volume.fix.replication) held the entire .dat resident and could OOM the process. Stream chunks through a bounded mpsc channel from a spawn_blocking reader instead; caps memory at ~16MB per transfer with backpressure.
* fix(volume): stream volume_incremental_copy from disk instead of buffering
Same buffering pattern as copy_file: every 2MB chunk was pushed into a Vec and only then returned via tokio_stream::iter, holding the entire delta resident. Stream the byte range from an owned file handle through a bounded mpsc channel, mirroring the copy_file fix.
* test(volume): cover streaming copy_file and volume_incremental_copy
Adds a multi-chunk .dat fixture and tests asserting both handlers stream in 2MB chunks (multiple messages), reassemble byte-for-byte, carry modified_ts_ns only on the first copy_file message, and honor stop_offset.
* address review: use u64 byte counters; stream local incremental copy without holding the store lock
- copy_file/volume_incremental_copy: track remaining bytes and offsets as u64 instead of casting uint64 stop_offset/dat_size through i64 (CodeRabbit).
- volume_incremental_copy: for local volumes open the .dat and stream directly with no lock held; only remote/tiered volumes take the per-chunk read_dat_slice path, so a remote S3 read is never performed while holding the store read lock (Gemini).
* volume (Rust): stream tiered incremental copy off the store lock, open .dat under it
Capture the reader for volume_incremental_copy while the volume lookup is still
under the store read lock: an open File for local volumes, a cloned remote
backend handle for tiered ones. Then drop the lock and stream with none held.
Opening under the lock pins the reader to the volume that exists now, so a
concurrent delete/recreate can't stream from the wrong file, and a slow S3
fetch for a tiered .dat no longer blocks store writers (the remote path
previously re-took the store lock per chunk).
Use a non-uniform copy-test payload so chunk reassembly catches duplicated or
reordered chunks a repeated byte would hide.
* volume (Rust): return empty when incremental-copy start offset is past the .dat
A corrupt needle index could locate an offset beyond the captured .dat size,
underflowing the dat_size - start_offset subtraction (panic in debug, wrap in
release). Guard it up front like the other empty-delta early returns.
---------
Co-authored-by: adri <adri@digitalunited.net>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Drop max-parallel so the 13 per-platform builds run together instead of two
waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late).
Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is
sha-independent, so it caches across releases and stops being the ~16-min
long-pole that gates the merge fan-in. go-build variants stay mode=min.
docker release: build per-platform on native runners, drop mode=max cache
The build job built every platform of a variant on one runner, so 2-4 Go
cross-compiles fought over a single 2-vCPU box and arm64 ran in an emulated
context. Split the matrix to one platform per job on a native runner
(amd64/386 on ubuntu-latest, arm64/arm-v7 on ubuntu-24.04-arm); only arm/v7
still needs QEMU, and only for its final apk stage. Each job pushes by
digest, and a new merge job assembles the multi-arch tag with imagetools
and mirrors it to Docker Hub.
cache-to mode=max -> mode=min: BRANCH=sha cache-busts the heavy go-build
layer every release, so writing all intermediate layers to the gha backend
spent 3-11 min per variant on a cache the next release's sha can never hit.
* test: add self-contained S3 read/write load tool
Concurrent PUT/GET against the S3 gateway, reporting requests/sec,
transfer rate, and latency percentiles. Built on the aws-sdk-go-v2
client the S3 tests already use, so no extra benchmark binary is needed.
* ci: add performance workflow
Three parallel jobs: cpu/heap pprof of the server under write load,
native throughput via weed benchmark plus the Go micro-benchmarks, and
an S3 read/write benchmark against the gateway. Runs on push to master
and manual dispatch with tunable duration, object count, size, and
concurrency.
* sts: enforce session-policy explicit deny during role chaining
A chained AssumeRole caller authenticates with an STS session token whose
inline session policy can explicitly deny sts:AssumeRole. The deny check only
evaluated the caller's named policies, so such a session could still chain into
any role its trust policy admits. Validate the session token in the deny check
and honor an explicit Deny in the inline session policy too.
* test(sts): integration coverage for AssumeRole authorization
Add an end-to-end AssumeRole authorization test (real weed mini + boto3):
a non-admin caller assumes a role its trust policy admits, an explicit
identity-side deny is blocked, and a session policy's explicit deny blocks
role chaining.
* sts: skip OIDC tokens and reject revoked sessions in the chaining deny check
Review follow-ups on the session-policy deny check:
- Guard session validation with !isOIDCToken so a bearer token our STS service
cannot validate does not error into a false deny.
- Reject a revoked session before evaluating its policy, restoring the
revocation enforcement the AssumeRole path lost when it stopped routing
through IsActionAllowed.
* fix(sts): authorize AssumeRole by the role's trust policy
The role's trust policy already declares who may assume it, but the caller
also had to pass an identity-side sts:AssumeRole check that only the Admin
action could satisfy — legacy static identities have no way to express
sts:AssumeRole on a role. So assuming any role required a full admin
identity. Drop the redundant check and let the trust policy be the authority;
scope it to specific principals to restrict who can assume.
* sts: resolve caller principal ARN for the trust-policy check
A legacy static identity can reach AssumeRole without a PrincipalArn set;
passing the empty value would miss a trust policy that names a concrete
principal. Resolve it to the canonical user ARN, sharing the logic
GetCallerIdentity already used inline.
* sts: enforce explicit identity-side deny for AssumeRole
Authorizing a named role by its trust policy alone dropped identity-side
evaluation entirely, so a caller whose attached policy explicitly denies
sts:AssumeRole could still assume any role the trust policy admits. Re-check
the caller's policies through the IAM manager for an explicit deny
(deny-always-wins) without requiring an allow; the trust policy stays the
allow authority.
* fix(postgres): prevent uint32 underflow & OOM in message parsing
* postgres: drop redundant startup guard, use maxStartupMessageSize const
The msgTotalLen < 8 check already guarantees msgLength >= 4, so the extra
msgLength < 4 guard before reading the protocol version was unreachable.
Point the startup size limit at maxStartupMessageSize instead of a literal.
* postgres: trim query terminator safely, cap pre-auth payloads
Use strings.TrimSuffix for the simple-query null terminator so a
non-null-terminated body isn't silently shortened, matching the auth
handlers. Bound password/MD5 reads with a dedicated maxAuthMessageSize
(10 KiB) instead of the 100 MiB maxMessageSize, since these payloads are
read before authentication.
---------
Co-authored-by: shangshuhan <shangshuhan@cmict.chinamobile.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
A FUSE interrupt is not a process kill. Go's async preemption (SIGURG)
makes a close() under load emit an interrupt on nearly every flush, so
deriving the metadata-flush context from the FUSE cancel channel turned
healthy concurrent close()s into EIO: the interrupt cancelled the
in-flight CreateEntry, which surfaced as "input/output error".
Bound the flush with a deadline instead. A healthy CreateEntry finishes
in well under a second, so the deadline only fires against a genuinely
stuck filer -- still keeping close() from hanging forever -- while
benign preemption no longer aborts a good flush.
Introduce security.BearerPrefix ("Bearer ", RFC 6750) and use it
everywhere an "Authorization: Bearer <token>" header is constructed,
replacing the scattered "BEARER "/"Bearer " string literals. SeaweedFS
matches the scheme case-insensitively when parsing (security.GetJwt), so
behavior is unchanged; this removes the magic string and settles the
casing on the standard form. The parser's upper-case comparison stays as
is on purpose.
The /?proxyChunkId= endpoint forwards the caller's headers to the volume
server but never mints a read token, so proxied chunk reads return 401
once jwt.signing.read.key is configured. Generate a fileId-scoped volume
token the same way the direct filer read path does, which fixes
filer.sync, filer.backup, filerProxy mounts, the MQ broker and the upload
gateway in one place.
* build(deps): bump com.fasterxml.jackson.core:jackson-databind
Bumps [com.fasterxml.jackson.core:jackson-databind](https://github.com/FasterXML/jackson) from 2.18.6 to 2.22.0.
- [Commits](https://github.com/FasterXML/jackson/commits)
---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
dependency-version: 2.22.0
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): pin jackson-annotations to its own 2.22 version
jackson-annotations dropped the patch digit in 2.20 and releases on its
own line, so 2.22.0 does not exist. Sharing jackson.version broke
dependency resolution; give it a dedicated property.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: replicate by fanning out from the gateway to every holder
The S3 gateway uploaded each chunk to one volume server, which then
relayed the copies to the other replica holders. The gateway now uploads
each chunk to every holder in parallel (type=replicate), removing the
primary volume server's receive-then-resend relay.
AssignVolume returns every replica holder (new repeated Location replicas,
forwarded from the master assign), the s3api captures them, and the
chunked uploader fans out whenever a chunk has more than one holder.
Cipher uploads keep the server-driven path since per-call encryption would
diverge the replicas.
* s3: cancel sibling replica uploads on the first failure
* s3: trim replica fan-out comments
* s3: roll back successful fan-out chunk copies when a holder fails
A failed fan-out records no FileChunk, so copies that landed on the holders
that finished before the cancel were leaked as orphans the caller could not
see. Track the holders that succeeded and delete the needle from each
(type=replicate, local-only) on failure, leaving nothing behind.
fix(wdclient): prevent stale cache fallback for empty volume locations
## Problem
During Kubernetes pod restarts, volume servers temporarily disconnect and their
locations are removed from vidMap. The deleteLocation function leaves an empty
array [] in vid2Locations map instead of removing the key entirely.
GetLocations() was checking 'if found && len(locations) > 0', which would fail
for empty arrays and fall back to the cache chain, returning STALE locations
from before the restart. This caused S3 gateway to try connecting to old pod
IPs that no longer exist, resulting in connection timeouts and hanging registry
sync jobs.
Example timeline:
1. Volume pod at 10.131.1.28:8081 registers volumes 10,12
2. S3 gateway caches: vid2Locations[10] = [10.131.1.28:8081]
3. Pod restarts, gets new IP 10.131.1.65:8081
4. Master sends delete → vid2Locations[10] = [] (empty, but key exists)
5. BUG: GetLocations(10) sees found=true, len=0 → falls back to cache
6. Returns stale 10.131.1.28:8081 instead of waiting for new location
7. S3 requests timeout trying to reach unreachable old IP
## Solution
Distinguish between two cases:
- found=true, locations=[] : Volume explicitly has no locations (e.g. restart)
→ Return nil, false (no fallback to cache)
- found=false : Volume never seen in current map
→ Check cache (preserve cache benefits for unknown volumes)
An empty array explicitly means 'this volume currently has no locations',
which is semantically different from 'volume unknown'. Don't fall back to
stale cache for explicitly empty volumes.
## Testing
Added comprehensive tests:
- TestGetLocationsEmptyArrayNoFallback: Verifies empty arrays don't use cache
- TestGetLocationsUnknownVolumeUsesCache: Verifies unknown volumes still use cache
- All existing tests pass
## Impact
Fixes registry sync job hangs during SeaweedFS upgrades/restarts. S3 gateway
will now correctly wait for updated volume locations instead of using stale
cached IPs.
Related: OutSystems.SeaWeedfs Helm chart, vega cluster incident 2026-06-24
A config-file reload (SIGHUP) routed through MergeS3ApiConfiguration,
which skips identities marked static so dynamic admin/filer updates can't
clobber them. That also blocked the config file itself from updating its
own identities, so editing a secretKey and reloading had no effect.
Thread a fromStaticFile flag from the file-load path into the merge: the
authoritative file overwrites its static identities (and reapplies service
accounts under them), while dynamic updates still leave them immutable.
Mark the rebuilt identities static in the merge so a concurrent
RemoveIdentity never observes them as removable mid-reload.
Standalone weed s3 created a master client and registered the receiving
SeaweedS3IamCache gRPC service, but never wrapped its credential store
with the propagating store. Only the filer-embedded path called
SetMasterClient, so IAM mutations on one s3 pod never reached peers; they
served a stale in-memory identity cache and returned InvalidAccessKeyId
until restarted.
Wrap the credential store with the master client when one is available,
mirroring the filer path, so mutations fan out over the existing gRPC
cache service.
* s3tables: allow hyphens in namespace and table names
Iceberg REST clients routinely use hyphenated namespace/table names, but the
S3 Tables charset (a-z, 0-9, _) rejected them with 400. Accept '-' as an
interior character (names must still start, and namespaces end, with a letter
or digit), making the catalog conformant for those clients. A permissive
superset of the AWS S3 Tables charset.
* s3tables: allow hyphens in table ARN parsing too
The ARN regexes still excluded '-', so parseTableFromARN rejected ARNs with
hyphenated namespace/table names and existing reject-the-hyphen tests broke.
Widen the ARN patterns to match the validator, retarget those tests at a
still-invalid leading-hyphen name, and cover ARN parsing with hyphens.
* s3tables: purge decoupled table data without deleting the reused name path
A renamed or created-over-leftover table keeps its data at a location that
differs from its catalog name path. Drop now purges that data location and
clears the marker, instead of recursively deleting the name path, which may
still hold another table's data.
* iceberg: route a table created over a leftover to a unique location
When the default location is occupied by a leftover directory (data kept when
another table was renamed to this name), create the new table at a unique
location so it cannot overwrite that table's metadata. Common case is unchanged.
* iceberg: fail table create when the leftover-path check errors
A transient filer lookup error fell through as "not occupied", routing the
new table back to the default path and risking the very overwrite this check
guards against. Propagate the error and return 500 instead.
* s3tables: assert all catalog xattrs cleared on decoupled drop
Seed the full marker set so the test catches a regression that leaves the
policy, tags, version, or entry-type attribute on the reused name path.
* s3tables: refuse to drop a table whose data path is an ancestor
Corrupt metadata can resolve the data path to the bucket or namespace root,
which the bucket-scope check still admits; a recursive purge there would wipe
sibling tables. Reject an ancestor data path before deleting.
A directory rebuild wiped the cached children, listed the filer once, and
published the directory authoritatively cached over whatever came back. A
transient empty listing -- a momentary list-stream glitch that ends as a
clean EOF with no entries -- then stranded a populated directory cached
over an empty store, hiding every file in it until some unrelated event
happened to rebuild it: stat returns ENOENT and readdir returns nothing
though the files are safe on the filer, and nothing re-triggers a build.
Re-read the directory when the listing comes back empty before trusting
it. The first re-read is immediate, since the likely transient clears on a
fresh stream; later attempts space out. A genuinely empty directory still
lists empty every time and caches as before, so only empty listings pay
the extra read.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
With default_permissions (the mount default) the kernel enforces unix
permission bits from the getattr/lookup attributes before it ever calls
Open, Create, or Mknod. The mount was re-checking permissions in
AcquireHandle and createRegularFile anyway, which duplicated the kernel's
work and kept the supplementary-group lookup on the per-file hot path.
Gate only the mode-bit access check on default_permissions being off, so
a non-root copy does no permission work on open/create. createRegularFile
still loads the parent to validate it exists, since the create RPC skips
the filer-side parent check. With default_permissions off the mount
remains the sole enforcer, so the full check still runs.
* operation: bound AssignVolume with a deadline
AssignVolume ran on context.Background(), so when the filer is overwhelmed
the RPC could block indefinitely and wedge every caller holding the
connection. Give it a 30s deadline so a stuck assign fails and the caller's
retry/error path runs instead of hanging forever.
* mount: abort flush when the FUSE request is interrupted
On close(), a killed process blocks in fuse_flush waiting for the mount to
answer. doFlush ran its metadata CreateEntry on context.Background() and
ignored the kernel interrupt channel, so against an overwhelmed filer the
flush never completed and the process stayed in uninterruptible sleep --
making the pod un-killable.
Derive a context from the FUSE cancel channel in Flush/Fsync and thread it
through doFlush -> flushMetadataToFiler -> streamCreateEntry; the retry loop
stops as soon as the context is cancelled. Release and the pre-rename flush
keep a non-cancellable context since they must finish regardless.
* operation: harden the AssignVolume timeout test
Make the test double's signal send non-blocking and bound the receive with a
timeout so a regression can't wedge the test instead of failing it.
* fix(filer.backup): repair chunk-incomplete and stale destination entries
filer.backup left destinations diverged while metadata advanced — chunk-incomplete
(missing/gapped ranges at full attr.file_size) or holding a chunk superseded by a
missed overwrite. The skip/repair decision keyed on filer.FileSize (the attr),
which a truncated entry keeps full, so it never repaired.
Decide from actual chunk state instead:
- coversReference: range-by-range containment (scalar byte totals and attr
FileSize/Md5 cannot see chunk-level gaps).
- hasStaleBackupChunk: a backup-written chunk (SourceFileId) the source no longer
lists; ignores out-of-band (rsync/direct) chunks.
- destinationMatchesReference: allocation-free positional fast path gating the
above so they run only on divergence (the in-sync path stays cheap).
- A strictly-newer destination is never repaired, so an older out-of-order replay
cannot roll it back. The stale signal is deferred at equal mtime (same-second
versions cannot be ordered; reliable S3 sub-second ordering is a separate fix).
Tests in filer_sink_test.go.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* filer.backup: verify chunk range in destinationMatchesReference fast path
The allocation-free fast path matched a destination chunk to its reference
by SourceFileId alone. That is correct today only because replicateOneChunk
copies the source chunk's Offset/Size verbatim, so SourceFileId identity
implies an identical range — an invariant that lives in another file with no
guard linking the two. If replication ever re-chunks (split/coalesce), a
chunk with the right SourceFileId but a different range would fast-path as a
full match and skip a needed repair (a false positive in the very class this
change otherwise prevents).
Compare Offset/Size alongside SourceFileId so the fast path is self-contained
and can only be more conservative (a range mismatch falls through to the
precise coversReference/hasStaleBackupChunk checks). Add tests for a shifted
offset and a larger size at matching identity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(filer): apply -filer.disk default to metadata log assigns
Metadata event log writes call operation.Assign directly and used only
FilerConf path rule DiskType. When filer.conf rules were missing or
unmatched, the master received an empty DiskType and grew volumes on the
built-in hdd layout.
Mirror resolveAssignStorageOption: wire FilerOption.DiskType into the
Filer, fall back when the matched path rule has no disk type, and return
the matched rule from resolveMetadataLogAssignDiskType to avoid duplicate
MatchStorageRule lookups.
Co-authored-by: Cursor <cursoragent@cursor.com>
* mini: fall back to -volume.disk for filer default disk type
weed server copies -volume.disk into the filer disk default when
-filer.disk is unset; weed mini did not, so metadata-log assigns sent
an empty disk type on clusters that only tag volumes (e.g. hot/warm).
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The tyler-smith/go-bip39 repository was deleted from GitHub, so go mod
download fails for anyone resolving it directly (GOPROXY=direct). It
only reaches us transitively through rclone's internxt backend, which
calls IsMnemonicValid and NewSeed. Point it at cosmos/go-bip39, an
API-compatible and maintained fork.
handleCreateTable used a type assertion that fails through WithFilerClient's
'all filers failed' wrap, so a concurrent create that the pre-check missed
fell through instead of returning the existing table. Use errors.As.
* fix(shell): correct volume.list -writable filter unit and comparison
* fix(shell): correct volume.list -writable filter unit and comparison
* chore(shell): fix typo in EC shard helper param names
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers
* fix: avoid reading upload body when writing JSON errors
* s3tables: add RenameTable operation
* iceberg: support table rename
* iceberg: test table rename
* s3tables: keep table data in place on rename
rename is catalog-only: drop the source's catalog xattrs in place instead of recursively deleting its directory, which wiped the metadata.json and data files the renamed destination still points at. treat a missing table-metadata xattr as NoSuchTable in GetTable so the soft-deleted source name stops resolving.
* s3tables: test rename preserves data
make the in-memory filer honor recursive data deletion and seed the source table's metadata/ and data/ children, then assert a rename leaves them intact, the source name resolves to NoSuchTable, and the destination resolves to the preserved location.
* iceberg: map rename errors through wrapped manager error
* s3tables: authorize rename destination namespace
rename moved a table into the destination namespace after only checking the source, letting a source-authorized caller place tables in namespaces they don't control. require CreateTable on the destination namespace and bucket before writing.
* s3tables: purge renamed table data on drop
* s3tables: test table data dir derivation
s3: don't write 503 to a disconnected client during remote cache wait
When the remote-only cache poll returns without chunks, re-check the
request context before emitting 503 + Retry-After. A client that
disconnected during the wait surfaces as context.Canceled, which the
caller already handles silently; writing to the closed connection only
produced broken-pipe log noise.
* s3tables: tag table entries and exclude views from table listings
* s3tables: add view CRUD operations
* iceberg: support view create, load, exists, drop, and list
* iceberg: support view update
* iceberg: test view error classification and metadata round-trip
* iceberg: pre-check existence and write view metadata only after create
* iceberg: map view namespace-not-found to 404
* iceberg: test view create namespace-404 and duplicate no-clobber
* s3tables: tag view metadata and entry type atomically
CreateView wrote ExtendedKeyMetadata and ExtendedKeyEntryType in two
UpdateEntry calls, so a partial failure could leave a view directory
untagged. Add setExtendedAttributes to set both in one UpdateEntry.
* iceberg: roll back view registration when metadata write fails
The metadata file is written after the catalog registers the view. If
that write fails, drop the just-created view so it doesn't linger
pointing at a missing metadata.json. Reuse the DeleteView path via a
shared dropView helper.
* iceberg: support multi-table transaction commit
Add handleCommitTransaction for POST /v1/transactions/commit. Validation
is atomic across all table-changes (resolve, load, evaluate every
requirement before any write); metadata writes and pointer flips are
best-effort with rollback, so this is not crash-atomic.
* iceberg: route transactions/commit with and without prefix
* iceberg: test transaction commit request decoding
* iceberg: restore full prior table state on transaction rollback
* iceberg: test transaction rollback restores full prior table state
* iceberg: only clean up metadata for rolled-back tables
* s3tables: add RegisterTable op
* iceberg: support table register
* iceberg: test register table
* iceberg: parse engine-written metadata version from location
* iceberg: test metadata version parsing for both filename forms
* iceberg: map register errors through wrapped manager error
* iceberg: validate register metadata-location bucket and reject traversal
* iceberg: log register metadata load failure
* filer: treat a directory carrying object data as an S3 key object
A file promoted to a directory by a child write keeps its chunks, inline
content, or remote-tiered entry. Recognize that as a directory key object,
not only when a Mime is set, so the object still lists, demotes on delete,
and is not reclaimed by cleanup like the object it still is.
* filer: keep the empty-folder cleaner from reclaiming a promoted object
The cleaner skips directory key objects, but its check only looked at the
Mime. Mirror the chunks/content/remote check so a file promoted to a
directory is not deleted once its children are gone.
* s3: serve ranged GET for a directory that carries object data
Reject only zero-size directories so a file promoted to a directory streams
range requests instead of returning 404, while empty directories still 404.
* s3: return HEAD metadata for a directory that carries object data
HEAD now 404s a directory only when it has no data, so a promoted object is
retrievable while empty/implicit directories still fall back to LIST.
* feat: add collection.mark shell command
Add collection.mark to mark all existing normal volume replicas in a collection as readonly or writable. The command runs in preview mode by default and requires -apply to execute changes. It reuses existing volume mark RPCs, supports default collection aliases, skips EC shards, and adds unit tests for option parsing and target collection logic.
* Revert "feat: add collection.mark shell command"
This reverts commit 50c2bbf94c.
* feat: support marking volumes by collection
Add a -collection option to volume.mark so operators can mark every normal volume replica in a collection using existing topology data and volume mark RPCs.
The change keeps the single-volume path unchanged and adds tests for collection target selection, EC shard exclusion, and argument validation.
Co-authored-by: Codex <noreply@openai.com>
* volume.mark: reuse eachDataNode for collection traversal
* volume.mark: continue past per-volume failures and report progress
Collection marking aborted on the first failed RPC, leaving the
collection half-marked with no record of which volumes succeeded.
Mark every reachable volume, print per-volume progress to the writer,
and return an aggregated error naming the failures.
* volume.mark: let -collection _default target the unnamed collection
Other volume commands use the _default sentinel to match volumes with
no named collection; volume.mark could not reach them at all. Map
_default to the empty collection name in the filter.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* worker: don't leak task goroutines on forced shutdown
Stop() drains in-flight tasks for 30s, then terminates the manager loop.
A task still running past that deadline later reports completion through
w.cmds - getAdmin in completeTask, the ActionIncTask* send, removeTask -
but with the loop gone and cmds unbuffered, those sends and the response
reads behind getAdmin/getTaskLoad block forever, leaking the goroutine.
Close a done channel when the loop exits and route the task-goroutine
sends through it so they abort and return zero values instead of
blocking. getAdmin can now return nil mid-shutdown, so collapse its
double-call sites to a single nil-checked call to avoid a deref.
* worker: abort remaining manager-loop sends after shutdown
Extend the post-shutdown abort to the sends that still blocked: Stop()'s
own ActionStop (so a second Stop, e.g. an admin-shutdown timer racing an
explicit one, doesn't hang), setTask, and handleTaskCancellation. Route
them through w.done so they return instead of blocking when the loop is
gone. Stop is now idempotent.
* fix(chunk_cache): close data/index files on initialization error
* chunk_cache: assign outer err on the .dat open path
The error-path defer keys off the function-level err, but the .dat
OpenFile used := and shadowed it, so that path relied on nothing being
open yet rather than the cleanup invariant. Assign the outer err so
every error return is uniform.
* chunk_cache: verify descriptor closure on POSIX, not just Windows
os.Remove succeeds on open files on Linux/macOS, so the removal check
only proved closure on Windows. Compare the open-fd count before and
after the failed load; gate the removal check to Windows.
---------
Co-authored-by: Contributor <contributor@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* 修复weedfs_xattr.go 中 XATTR_REPLACE 语义缺失
* mount: fix XATTR_CREATE/XATTR_REPLACE flag semantics in setxattr
XATTR_CREATE fell through into the XATTR_REPLACE branch: creating a new
attribute hit the empty-oldData guard and returned ENODATA instead of
creating it, while creating over an existing attribute silently succeeded
without the EEXIST that setxattr(2) requires. Drop the fallthrough chain
so CREATE returns EEXIST when the attribute already exists, REPLACE
returns ENODATA when it is missing, and otherwise the value is written.
Test existence via the map lookup so an attribute with an empty value is
still treated as present.
---------
Co-authored-by: 王郁文 <wangyuwen@cmict.chinamobile.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix: Resolve inconsistent usage of error variables
* mysql2: guard nil DB on open failure and wrap connect error
---------
Co-authored-by: muminglei <muminglei@cmict.chinamobile.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* iceberg: return 400 for invalid namespace/table names
The S3 Tables name charset (a-z, 0-9, _) is stricter than the Iceberg
REST spec, so clients sending hyphens or uppercase hit a validation
error. That error fell through to 500; it's client input, so map it to
400 BadRequestException across the namespace and table handlers.
* iceberg: tighten name-validation error matching
Match the validator's own phrasings (invalid/must/cannot) instead of a
bare "namespace name"/"table name" substring, so an unrelated fault that
happens to mention a name isn't misreported as a 400. Lowercase first to
stay robust to message capitalization.
* iceberg: support namespace property updates
Add POST /v1/namespaces/{namespace}/properties to the REST catalog. It
applies the request's removals and updates and returns the removed/updated/
missing summary the spec defines. A new UpdateNamespace op on the S3 Tables
manager rewrites the stored namespace properties; AWS S3 Tables namespaces
have no properties, so this is the SeaweedFS-side backing for the catalog.
* iceberg: dedup namespace property removals
A key repeated in removals was deleted on its first occurrence, then
reported as missing on the next — landing in both removed and missing.
Skip keys already processed.
* iceberg: map namespace-update backend errors to REST statuses
UpdateNamespaceProperties returned 500 for every manager failure, masking
the namespace being dropped between read and write, or a denied caller.
Inspect the typed S3TablesError and answer 404/403 accordingly, 500 only
for the rest. Also replaces the GetNamespace not-found string match.
* iceberg: test the namespace-properties conflict path
Cover the 422 returned when a key appears in both removals and updates.
The check runs before any backend call, so it needs no filer.
* fix(s3api): preserve requested AES256 copy encryption
Problem
CopyObject metadata processing ignored an explicit x-amz-server-side-encryption: AES256 request header. A destination copy could lose the requested SSE-S3 metadata even though KMS requests were handled.
Root cause
processMetadataBytes only wrote the destination SSE header when the requested algorithm was aws:kms. Any other explicit SSE algorithm fell through to the source-preservation branch.
Fix
Write the requested SSE algorithm whenever x-amz-server-side-encryption is present, and keep KMS-specific metadata handling limited to aws:kms.
Co-authored-by: Codex <noreply@openai.com>
* fix(s3api): reject unsupported copy encryption algorithms
A mistyped or unsupported x-amz-server-side-encryption value on a copy
request slipped past validation and got persisted as the destination's
algorithm header, advertising encryption that was never applied. Reject
anything other than AES256 or aws:kms up front.
* fix(s3api): write SSE key metadata for empty encrypted copies
A zero-byte source copied with an explicit SSE request took the
no-content branch and never ran the encryption path, leaving the object
with a bare algorithm header but no key. HEAD then advertised SSE while
the encryption-state machine saw the header as orphaned. Run the inline
encryption path when the destination requests encryption so the key
metadata is written too.
* s3api: use SSEAlgorithmKMS constant in copy metadata handling
* test(s3api): cover source SSE preservation on copy
* test(iam): allow the local client's real source IP in SourceIp tests
The aws:SourceIp allow policies hardcoded the loopback CIDRs, but a CI
runner reaching the server over localhost can be observed with one of the
host's RFC1918 addresses (the S3 endpoint is advertised on a 10.x
interface), so the positive-condition PutObject was denied and the allow
assertion flaked while the deny path passed trivially. Broaden the allow
list to loopback plus private ranges via a shared helper, and log the
denial on each failed attempt so any residual failure is diagnosable.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
In ec_task.go, 23 fmt.Errorf calls used %v verb to wrap errors,
breaking the error chain introduced in Go 1.13. This prevents
callers from using errors.Is() and errors.As() to inspect the
underlying error type.
Changed all fmt.Errorf calls from %v to %w to properly wrap
errors, preserving the error chain for upstream callers.
Note: glog.* logging calls and fmt.Sprintf calls intentionally
keep %v as they are not error wrapping contexts.
Co-authored-by: 吴奇臻 <wuqizhen@cmict.chinamobile.com>
* build(deps): bump github.com/apache/iceberg-go from 0.5.0 to 0.6.0
Bumps [github.com/apache/iceberg-go](https://github.com/apache/iceberg-go) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/apache/iceberg-go/releases)
- [Commits](https://github.com/apache/iceberg-go/compare/v0.5.0...v0.6.0)
---
updated-dependencies:
- dependency-name: github.com/apache/iceberg-go
dependency-version: 0.6.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* iceberg: adapt worker to iceberg-go 0.6.0 API
Fields() now yields iter.Seq2 (index, value); SortField.SourceID and
PartitionField.SourceID are methods backed by SourceIDs; RemoveSnapshots
takes a postCommit flag (false here, file cleanup runs through the filer).
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(shell): correct volume.list -writable filter unit and comparison
* fix(shell): correct volume.list -writable filter unit and comparison
* chore(shell): fix typo in EC shard helper param names
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers
* fix(shell): exclude failed EC shard copies from rebuild recoverability gate
prepareDataToRecover incremented the remote-shard counter before the copy
RPC, so in apply mode a failed VolumeEcShardsCopy was still counted toward
the DataShardsCount recoverability gate. The gate could then pass with
fewer real shards than required, deferring the failure to the deeper
generateMissingShards/reconstruct step and reporting an inflated shard
count in the "not enough shards" error.
Count the remote shard only after a successful copy (apply mode) or when
planning (dry-run), and rename wouldCopy to recoverableRemoteShards for
clarity. Add a regression test covering an apply-mode copy failure.
* fix(shell): clean up copied EC shards when the recoverability gate fails
A runtime copy failure can trip the gate after earlier copies already
succeeded, stranding those working shards on the rebuilder. Return the
copied shard ids on the error path and run the cleanup defer even when
recovery fails, so the temp shards get deleted.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(filer): propagate proxyChunkId query params to volume server
When weed mount reads via filer proxy mode (-volumeServerAccess=filerProxy),
the mount adds query params like readDeleted=true to chunk read requests.
Two bugs prevented these from working:
1. filer_server_handlers.go extracted fileId from the raw RequestURI, which
includes query params, corrupting the fileId (e.g. '6,abc&readDeleted=true').
Fix: use r.URL.Query().Get("proxyChunkId") for clean extraction.
2. filer_server_handlers_proxy.go didn't forward query params to the volume
server. The urlStrings from LookupFileId already contain the fileId in the
path, so just append the original query string.
* filer: match chunk proxy by query param, not URI prefix order
Order-dependent prefix slicing missed proxyChunkId when it wasn't the
first query param. Gate on root path and read the parsed query value.
* filer: drop internal proxyChunkId from proxied volume query
Lookup URLs already carry the fileId in the path, so forwarding the raw
query duplicated proxyChunkId onto the volume server. Strip it and only
append the remaining caller params (e.g. readDeleted).
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
util/http: lazily init the global HTTP client
GetGlobalHttpClient returned a nil client until InitGlobalHttpClient ran,
which only happens in weed.go's main. Anything that starts a command
in-process bypasses that: the admin server's metrics goroutine seeds a
dashboard sample on startup, reaching fetchPublicUrlMap -> GetGlobalHttpClient().Do,
and nil-derefs the receiver in GetHttpScheme.
Init the client on first Get via sync.Once so it is never nil regardless of
the startup path. InitGlobalHttpClient keeps its eager-init role through the
same Once.
prometheus/common v0.67.5, golang.org/x/sys v0.45.0, klauspost/compress
v1.18.6 to match the root module; readonly build was rejecting the stale
versions.
chore(deps): bump cassandra-gocql-driver to v2.1.2
Picks up upstream fixes for the Cassandra filer store: HostFilter panic
when a keyspace isn't replicated to every DC, broken system.peers
fallback, and pool connection errors under a small Session.Timeout.
* fix(s3api): sort repeated SigV4 query values
Problem
SigV4 canonical query strings must sort repeated parameter values. SeaweedFS preserved the incoming value order, so a correctly signed request with repeated query parameters could fail verification when the request order differed from canonical order.
Root cause
getCanonicalQueryString delegated directly to url.Values.Encode(), which sorts parameter names but preserves each key's value slice order.
Fix
Sort every query value slice before encoding the canonical query string, while still excluding X-Amz-Signature for presigned URLs.
Reproduction
go test ./weed/s3api -run TestGetCanonicalQueryStringSortsRepeatedValues -count=1 failed before the fix with partNumber=2 before partNumber=10.
Co-authored-by: Codex <noreply@openai.com>
* test(s3api): expand canonical query coverage
Co-authored-by: Codex <noreply@openai.com>
---------
Co-authored-by: Codex <noreply@openai.com>
* fix(shell): correct volume.list -writable filter unit and comparison
* fix(shell): correct volume.list -writable filter unit and comparison
* chore(shell): fix typo in EC shard helper param names
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers
* feat(shell): support batched EC encode and multi-volume selection
Add -volumeIds (comma-separated) and -batchSize flags to ec.encode.
When -batchSize > 0, volumes are processed in independent batches, each
committed separately: encode -> rebalance -> verify -> delete originals.
This bounds the working set and lets source volumes be reclaimed without
waiting for the entire set to finish, at the cost of per-batch rebalancing.
Because each batch deletes its originals, a failure in a later batch is
unrecoverable for already-completed batches.
To let the single-volume, multi-volume, and collection paths share one
per-batch routine, the re-balance scope is now always derived from the
volumes actually selected for encoding (collectCollectionsForVolumeIds),
rather than every collection matching the -collection regex. Practical
effect: with -collection, a collection that matches the pattern but
contributes no encodable volumes is no longer re-balanced as a side effect.
The -volumeId path is unchanged; -batchSize=0 (default) preserves the
original single-pass flow.
The per-batch routine reuses the existing assertEncodableRegularVolumes
guard, doEcEncode skipped-node handling, and verifyEcShardsBeforeDelete
retry loop. The capacity pre-flight check takes the already-fetched
topology instead of issuing another VolumeList to the master per batch.
Also clarify the -collection flag description to note it accepts a regex
pattern, matching the existing command help.
-volumeId and -volumeIds are mutually exclusive; ids in -volumeIds are
validated and de-duplicated.
During leader election, MaybeLeader() returns empty string immediately (non-blocking),
causing master to return NotLeaderError without sending HeartbeatResponse.Leader.
Volume servers depend on HeartbeatResponse.Leader to discover the new leader address,
so they keep retrying the old leader and cannot switch.
Switching to Leader() restores the 20-second exponential backoff retry behavior,
ensuring volume servers receive the new leader address as soon as election completes.
* feat: add Prometheus metric for volume creation operations
Add VolumeServerVolumeCreationCounter metric to track volume creation
attempts by result (success/failure).
Changes:
- Add VolumeServerVolumeCreationCounter in weed/stats/metrics.go
- Instrument GrowByCountAndType in weed/topology/volume_growth.go
- Add unit tests in weed/stats/metrics_volume_creation_test.go
- Add Grafana dashboard panel for volume creation rate
This metric enables monitoring volume creation success/failure rates
in SeaweedFS clusters.
* fix: address CodeRabbit review - move failure counter before early return
- Move failure counter into loop else block before return statement
- Move success counter to after loop completion
- Strengthen test assertion from count < 1 to count != 2
- Add t.Cleanup() for test isolation in TestVolumeCreationCounterIncrement
* fix: count each volume create attempt and move metric to master subsystem
- topology runs on the master, so move volume_creation_total from the
volumeServer subsystem to master (SeaweedFS_master_volume_creation_total);
rename the var to MasterVolumeCreationCounter and group it with the other
master metrics.
- increment the counter per findAndGrow iteration instead of once per
GrowByCountAndType call: each logical volume is now counted, partial
successes before a failure are credited, and a targetCount==0 call no
longer records a spurious success.
- update the Grafana panel query and the unit tests; the registration test
now asserts via the shared Gather registry under the fully-qualified name.
* test: drop volume_creation metric tests
They only exercised the Prometheus client library (CounterVec increment and
registry collection), not any SeaweedFS behavior, so they carried maintenance
cost without verifying anything in this codebase.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* mini: resolve admin credentials from security.toml and env vars
weed mini started the admin UI without resolving admin.user/admin.password
(and the read-only pair) from security.toml [admin] or WEED_ADMIN_* env vars,
so the only way to protect the UI was the -admin.password flag. The standalone
weed admin command applies these fallbacks in runAdmin via applyViperFallback;
the mini path calls startAdminServer directly and skipped it, leaving
authRequired false and the UI unauthenticated.
* mini: load admin.toml maintenance settings
The mini admin path runs ApplyMaintenanceConfigFromToml (via startAdminServer)
against the global viper, but runMini never merged admin.toml, so file-based
maintenance task settings ([maintenance.vacuum], .balance, .erasure_coding)
were ignored under mini while the standalone weed admin honored them. Load it
alongside master/volume config.
* mini: support -admin.urlPrefix for the admin UI
Expose the reverse-proxy subdirectory prefix that the standalone weed admin
already supports, so the mini admin UI can run under e.g. /seaweedfs. The
prefix is normalized the same way and passed through to startAdminServer.
* feat: add Prometheus metrics for replication operations
Adds 5 metrics to instrument volume server replication (write/delete):
- Operations counter with success/failure labels
- Duration histogram for latency tracking
- Targets gauge for replica fanout
- Failures counter with error reason labels
- Under-replicated volumes gauge on master
* fix: record replication duration histogram only when replicaCount > 0
* fix: update replication targets gauge for all operations including zero
* fix: ensure symmetric replication success/failure counting and proper metrics updates
* fix: change VolumeServerReplicationTargets from Gauge to Histogram
- Replace .Set() with .Observe() in store_replicate.go (2 occurrences)
- Update test to use CollectAndCount for histogram assertion
- Rename TestReplicationTargetsGauge -> TestReplicationTargetsHistogram
- Update documentation to reflect Histogram type and PromQL examples
* Add comments to replication metrics and improve test coverage
* metrics: add replication panels to grafana dashboard
Master row gets an under-replicated volumes timeseries; Volume Servers
row gets replication operations, failures-by-reason, p99 duration, and
average fan-out panels for the new replication metrics.
* metrics: name the replication duration histogram replication_seconds
Match the volumeServer convention (request_seconds, vacuuming_seconds)
rather than the admin/lifecycle _duration_seconds spelling.
* metrics: guard replication fan-out panel against divide-by-zero
clamp_min the _count rate so the avg-targets ratio reads 0 instead of
NaN when there are no replication events in the window.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* mount: attach Content-MD5 to chunk uploads
Mount writes never set UploadOption.Md5, so FileChunk.ETag stays empty
and filer.ETag() degenerates to md5("")-N: same-size files compare
equal regardless of content, defeating metadata-level verification
like filer.sync.verify.
Compute each chunk's MD5 and send it as Content-MD5. The volume server
verifies it on ingest (rejecting in-flight corruption) and echoes it
back, persisting FileChunk.ETag like filer/S3 writes already do.
The dirty-page flush paths already pass a *util.BytesReader whose
backing slice is the whole chunk, so the digest is taken in place with
no extra read, copy, or allocation (UploadWithRetry unwraps it the same
way downstream). Only the rarer plain-reader callers (e.g. manifest
chunks) fall back to io.ReadAll. Skipped under -cipher, where only the
ciphertext reaches the server.
The digest encoding (std-base64 of the raw md5) is the contract the
volume server verifies against, so it is factored into contentMD5Base64
and covered by a unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* mount: compute chunk Content-MD5 in the uploader, not the caller
Move the WantMd5 hashing into UploadWithRetry, where the chunk is already
buffered for the retry path, so saveDataAsChunk stops type-switching the
reader and re-reading plain readers. One materialization point, and the
cipher exclusion lives next to the hash instead of at every call site.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume: detect phantom volumes held open as deleted FDs
Add disk-file validation in heartbeat collection to prevent reporting
phantom volumes that exist in memory but are deleted from disk. This
unblocks re-replication when files are unlinked while the volume server
holds them open via file descriptors.
Cache disk checks per-volume with 30-second TTL to avoid syscall overhead.
Implement in both Go and Rust volume servers.
* volume: make last_disk_check_ns field public for heartbeat access
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes if file_count > 0
Use file_count as the indicator for whether a volume held actual data,
rather than volume size. Phantom volumes only occur when a volume that
had files is deleted while the process holds open file descriptors.
Test volumes with no file count won't trigger the phantom detection check.
* volume: stat the .dat with its extension when detecting phantom volumes
DataFileName()/IndexFileName() return the extensionless base path, so os.Stat
saw every volume's files as missing and dropped it from the heartbeat, leaving
the master with no locations and breaking deletes/lookups. Stat FileName(".dat")
instead, skip remote-tiered volumes whose .dat lives in cloud storage, and
re-check a missing file every heartbeat rather than caching the negative.
* filer: skip COLLATE "C" list fallback on CockroachDB
CockroachDB string comparison is already byte-ordered, so wrapping the
list queries in COLLATE "C" can never change result order. It reports
datcollate=en_US.utf8 regardless of how the database was created, so the
collation check always misfires and forces the fallback. On 22.1 and
older COLLATE "C" is rejected as an invalid locale, turning every filer
list query into a hard failure. Detect the backend via version() and
keep the default ordering.
* filer: test CockroachDB collation detection
Cover the CockroachDB skip path and the locale-aware Postgres force path
with go-sqlmock.
* fix(mount): run entry invalidations off the meta-cache apply loop
The apply loop ran invalidateFunc inline, which acquires the open file
handle's lock in fhLockTable. Meanwhile flushMetadataToFiler holds that
same fh lock and then waits on the apply loop (applyLocalMetadataEvent).
When both target the same open file concurrently, the loop blocks on the
fh lock while the lock holder blocks on the loop: an ABBA deadlock that
backs up every later readdir/flush and hangs the mount.
Fix: dispatch entry invalidations to a dedicated FIFO worker goroutine so
the apply loop never blocks on locks held by goroutines waiting on it.
Adds a regression test reproducing the interleaving.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* perf(mount): update invalidate counter once per batch
Run the batch's invalidateFunc calls without re-taking invalidateMu per
item, then bump invalidateProcessed and broadcast once after the loop.
WaitForEntryInvalidations only needs the count to reach its target and a
batch always completes together, so the per-item lock + broadcast was
wasted work.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* mount: extract the invalidate worker into util.AsyncBatchWorker
The apply loop's off-thread entry-invalidation queue was a one-off mutex +
cond + slice + counters living inside MetaCache. Pull it out as a generic
unbounded FIFO worker so the deadlock-avoidance contract (never block the
producer, drain on shutdown, wait-for-quiesce) lives in one place.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
GetObjectStoreUserDetails only returned identity.PolicyNames (attached
managed policies) and omitted per-user inline policies. Inline policies are
stored separately from the identity record and are authoritative at S3
enforcement time (they take precedence over the legacy Actions list), so an
operator could not see what actually governed a user's access via the admin
API/UI.
Include inline policy names (via credentialManager.ListUserInlinePolicies) in
the returned PolicyNames. Adds a unit test using the memory credential store.
* mount: cache supplementary group IDs to improve non-root access performance
* mount: clear supplementary group cache between tests and add cache verification test
* mount: add docstrings and benchmarks for supplementary group cache
* mount: add performance test demonstrating cache effectiveness
* mount: add TTL-based cache expiry for supplementary group IDs (5-minute refresh)
* s3: add streaming reader interface for remote storage
Add RemoteStorageStreamReader optional interface to support efficient
streaming of large remote objects without buffering entire file in memory.
This enables future stream-through caching where data can be served to
clients while simultaneously writing to volume servers.
Implement ReadFileAsStream() for S3, GCS, and Azure backends using their
native streaming APIs. This provides the foundation for improving TTFB
on large remote file access by serving data directly from remote storage
while background cache operation populates local chunks.
The streaming interface allows remote storage backends to return io.ReadCloser,
enabling efficient memory usage for multi-GB objects compared to the
current ReadFile() approach which buffers entire ranges in memory.
* s3: adaptive timeout for remote object caching to improve TTFB
Use size-aware cache polling timeout to balance cache-hit rate against
time-to-first-byte:
- Small files (<50MB): 10s timeout - more likely to complete caching
before timeout, improving subsequent request performance
- Medium files (50-500MB): 5s timeout - default balance
- Large files (>500MB): 2s timeout - fail-fast to improve initial TTFB
for very large downloads
This reduces waiting time for large remote files while maintaining
high cache-hit rate for smaller files that cache quickly.
* s3: address code review feedback for stream-through cache
- Move startBackgroundRemoteCache call after policy recheck to avoid
cache side effects for denied requests (authorization first)
- Make startBackgroundRemoteCache version-aware by accepting versionId
parameter and using buildVersionedRemoteObjectPath
- Add timeout (5 minutes) to background cache context to prevent
goroutine pile-up if RPC stalls under load
- Update cacheRemoteObjectForStreamingWithShortTimeout to return both
entry and error, allowing callers to distinguish transient errors
(timeout/cancellation) from permanent errors (not found, denied)
- Update streamFromVolumeServers to handle permanent cache errors with
appropriate HTTP status codes (404 for not found, 503 for transient)
* helm: reject emptyDir for volume idx
An ephemeral index on a separate volume is wiped on every pod restart
while the .dat/.vif persist on the data PVCs. The volume server then
finds data with no matching .idx and exits via glog.Fatalf, putting the
pod into CrashLoopBackOff with no automatic recovery.
emptyDir is never the right choice for idx: if the data is persistent it
is a durability mismatch, and if the data is also ephemeral the default
(idx co-located with the data) already covers it. Fail the render with a
clear message pointing at the default or a persistent volume instead, and
drop emptyDir from the documented idx options.
* helm: rebuild a missing volume idx on restart
With emptyDir rejected, a separate idx volume is always persistent
(hostPath/PVC/existingClaim) -- but it can still lose its .idx out of
band (e.g. a node-local PVC reprovisioned on reschedule, or a pre-9944
compaction crash that left a .dat without its matching .idx). The
seaweedfs-vol-move-idx init container already moves idx files next to the
data into the index dir; have it first regenerate, via weed fix, any .idx
absent from both the data dir and the index dir, then move it into place.
The rebuild only runs when an idx is genuinely missing, so a healthy
index adds no startup cost.
* volume: validate remote S3 endpoints in FetchAndWriteNeedle (Rust)
Port the Go volume server's SSRF guard to the Rust volume server. The
gRPC FetchAndWriteNeedle reads from a caller-supplied S3 endpoint, so an
unguarded server can be pointed at loopback / link-local / RFC1918 /
CGNAT / cloud-metadata hosts to read internal services. Resolve and
reject those endpoints unless -volume.allowUntrustedRemoteEndpoints is
set (default off), mirroring weed/server/volume_grpc_remote.go.
Connect-time re-validation against DNS rebinding (Go's guardedDialer) is
not yet ported: the aws-sdk-s3 client builds its own connector, so the
up-front resolve-and-check leaves a narrow TOCTOU window. Left as a
follow-up.
* volume: harden remote endpoint guard (IPv4-mapped IPv6, all S3 types)
Address SSRF review feedback:
- Normalize IPv4-mapped IPv6 (::ffff:a.b.c.d) to IPv4 before the deny
checks, so ::ffff:127.0.0.1 / ::ffff:169.254.169.254 no longer slip
past the IPv4 rules.
- Validate the endpoint for every S3-compatible backend, not just type
"s3"; wasabi/backblaze/aliyun/... all dial a caller-supplied endpoint
through the same client. Skip validation when the endpoint is empty
(the provider default, e.g. real AWS S3, which cannot reach internal
hosts).
* volume: set allow_untrusted_remote_endpoints in integration-test state
The tests/http_integration.rs VolumeServerState literal was missed, which
broke cargo test compilation (it builds the integration tests, unlike the
cargo test --lib used locally).
EC bitrot protection is now a fixed default: always on at 16 MiB block
granularity. These volume-server flags exposed needless configurability;
the package defaults in erasure_coding (BitrotProtectionEnabled,
BitrotBlockSize) are retained and still drive sidecar generation. Drop the
now-unused erasure_coding import.
TopicExists can return a transient false-negative for a topic that is in
fact present (a broker/filer blip under load, or a just-created topic whose
existence cache is still stale). The metadata and produce handlers then tried
to auto-create, hit "topic already exists", treated that as a failure, and
dropped the topic from the Metadata response - so the client saw a spurious
UNKNOWN_TOPIC_OR_PARTITION right after the topic was created.
Return a sentinel ErrTopicAlreadyExists from the create paths and add
ensureTopicExists, which treats it as confirmation the topic exists. It also
folds in the repeated TopicExists -> invalidate -> recheck -> create logic
shared by every metadata handler (v0-v8) and both produce paths. v2+ produce
now auto-creates too, matching the auto.create.topics.enable=true behavior the
rest of the gateway already simulates.
Add path filters to workflows that fired on every PR/push regardless
of the diff: CodeQL, go build, the e2e/EC/vacuum/TLS/plugin-worker
integration suites, the Kafka and Postgres gateways, the S3 suites
(Ceph s3tests, s3-go, s3-tables, proxy-signature, https, example,
filer-group), TUS, and the dev binary/container builds. Each scopes
to its subsystem under weed/, its test dir, go.mod/go.sum, and the
workflow file, so docs-, helm-, terraform-, rust- or java-only
changes no longer trigger a full compile-and-test fleet.
* mount: apply cached remote entry without blocking the read
Reading an uncached remote-mounted file hung forever. The read holds the
file-handle shared lock across the on-demand download, then synchronously
waits on the metadata apply loop to apply the cached entry. The filer's
update event for that same object reaches the apply loop first and runs
invalidateFunc, which wants the file-handle exclusive lock — held in shared
mode by the still-running read. The loop blocks on the read; the read blocks
on the loop.
Enqueue the apply without waiting so the read never blocks on the apply loop
while holding the lock. The handle is already updated via SetEntry, and the
filer subscription delivers the same event regardless.
* mount: regression test for uncached remote read deadlock
Drives the real Read path through downloadRemoteEntry with a stub filer that
broadcasts the matching invalidate event before returning, reproducing the
lock-ordering deadlock. Fails (times out) without the fix.
* s3api: add optional request interceptor to circuit breaker
Add an optional Interceptor func(next http.HandlerFunc, action string) http.HandlerFunc
field on CircuitBreaker, applied at the very top of Limit() -- before upload
concurrency limiting and before the 'if !cb.Enabled' early return -- so it runs
for every route regardless of breaker state.
It is nil by default (no behavior change). An interceptor may reject a request
(write its own response and skip next) or wrap next to observe/shape it. This
provides a single per-request extension point at the existing per-route
chokepoint (cb.Limit already wraps nearly every S3 route), useful for tracing,
auditing, or request rate limiting without touching the route table.
* s3api: evaluate circuit breaker interceptor per request
Move the Interceptor nil-check into the returned handler so it is consulted
per request instead of captured at registration time. This:
- lets the interceptor be installed after registerRouter runs (handlers are
built during construction, before request-time dependencies exist), and
- avoids dereferencing a nil CircuitBreaker at registration time, which
panicked tests that register routes on a server with a nil cb
(e.g. TestRouting_STSWithQueryParams).
Adds a regression test for installing the interceptor after Limit() returns.
* fix(shell): return fs.verify topology errors
Problem: fs.verify can silently ignore master topology lookup failures before verifying files.
Root cause: commandFsVerify.Do returned parseErr after collectVolumeIds failed, but parseErr is nil once path parsing succeeds.
Fix: Return the actual collectVolumeIds error so VolumeList and master client failures stop the command.
Co-authored-by: Codex <noreply@openai.com>
* remove the tests
---------
Co-authored-by: Codex <noreply@openai.com>
isLocalOnlyEntry resolved the pinned-child check through inodeToPath. A kernel
Forget drops the path→inode mapping once the lookup count reaches zero, but an
async writeback flush — and the file handle, still in fhMap during the drain —
is keyed by inode and outlives that mapping. Between Release dispatching the
async flush and the flush reaching the filer, a Forget could unpin an in-flight
create, so a concurrent directory rebuild would wipe it and the file would
ENOENT until the flush lands and the cache refreshes.
Key the check off the inode the store entry already carries (createFile stamps
it into the placeholder), so the pin no longer depends on a mapping Forget can
remove.
* s3: bound streaming remote-cache wait so large cold GetObject returns 503, not a hang
A remote-only object whose download outlasts the 30s dedup attempt fell through
to cacheRemoteObjectForStreaming, which waited on the raw request context with no
bound. For multi-GB blobs the client read timeout fires first, so the request was
canceled and surfaced as InternalError rather than the 503 + Retry-After the
handler already emits. The filer keeps caching on a detached context, so the
retry would have streamed from the cached chunks.
Bound the wait under common client read timeouts: the 503 fires while the client
is still connected, and a retry picks up the finished cache.
* s3: make the streaming remote-cache timeout atomic
The bound is a package-level var so tests can shorten it. Hold it as an
atomic int64 so a test that stores a short value can never race the request
path that loads it.
* s3: assert the streaming-cache timeout test waits on the bound
The upper-bound-only check would also pass if the RPC returned immediately on a
setup error. Add a lower bound so the test fails unless the bounded wait fired.
wfs.Create defers the filer create to Flush and inserts a local-only
placeholder into the metaCache (dirtyMetadata=true), so a just-created file
exists locally before the filer has it. When the parent directory falls out of
cache (hot-dir read-through, idle evict) and is rebuilt, EnsureVisited wipes the
store and refills from a filer listing that does not yet include the un-flushed
create, then markCachedFn publishes the directory authoritatively cached without
it. lookupEntry then returns an authoritative ENOENT and ReadDir returns nothing
— the file disappears from the mount although the client created it. Under
concurrent read+write churn on one directory this is the ConcurrentReadWrite
flake.
Preserve children the mount flags as local-only (open dirty handle or pending
async flush — the signal lookupEntry already trusts) across the rebuild's wipe
instead of blind-deleting them. Unpinned stale children are still dropped so a
rebuild cannot resurrect a deleted entry.
The s3 and rclone tiered-storage backends were registered via blank imports
in weed/storage (volume_tier.go and volume_info/volume_info.go). That forced
every library consumer of weed/storage -- weed/shell, and through it external
tools -- to link aws-sdk-go and, under the rclone build tag, the full rclone
backend set and its cloud-storage SDKs, even though those consumers never tier
volumes.
Move the registrations into a new weed/storage/backend/all aggregator and
blank-import it once from weed/command, the binary's composition root. The weed
binary still registers both backends; weed/storage and its library consumers no
longer pull the backend SDKs into their dependency graph.
* SECURITY.md: require working repro and trust-boundary impact
Funnel reports through GitHub private vulnerability reporting, require a
reproduction against a supported version, and spell out the trust model so
internal-cluster exposure and example-config issues are out of scope.
Unvalidated scanner/AI output is handled as hardening, not as an advisory.
CVEs minted by third parties outside this process may be disputed.
* SECURITY.md: drop email channel, GitHub private reporting only
* mount: tolerance-window write pattern detection for concurrent writeback
WriterPattern decides whether each dirty chunk is buffered in RAM
(NewMemChunk) or spilled to an on-disk swap file (NewSwapFileChunk), so a
sequential stream misclassified as random is pushed through disk-backed
swap.
The old detector compared each write's start to the previous write's
exact stop offset (atomic.Swap of lastWriteStopOffset, then
lastOffset == offset). That is brittle under concurrent FUSE writeback:
MonitorWriteAt runs before the per-handle write lock, so parallel Write
upcalls interleave the swap, and writeback flushes dirty pages slightly
out of offset order. A genuinely sequential stream then repeatedly reads
as random and spills to swap.
Replace the exact match with the same frontier + tolerance approach used
on the read side: track a never-regressing max frontier (offset+size) via
a CAS loop and treat a write whose start is within SeqTolerance (8 MiB) of
that frontier as sequential. The existing +/-ModeChangeLimit hysteresis is
kept, so a single outlier write can't flip the mode.
Adds page_writer_pattern_test.go covering sequentiality, hysteresis,
reorder tolerance, the inclusive tolerance boundary, frontier
non-regression (the CAS invariant), and recovery back to sequential mode.
* mount: read write frontier inside the CAS loop
Capture the frontier from the CAS loop's own load rather than a separate
up-front snapshot, so the sequentiality diff is judged against the freshest
pre-image even if a concurrent writeback upcall advances the frontier while
we loop. Also drops the now-redundant load. Behavior is unchanged for the
single-threaded tests; addresses review feedback on #9984.
* filer: tolerance-window read pattern detection for concurrent readahead
ReaderPattern detected sequential access via strict contiguity (lastReadStopOffset
== offset). Under concurrent/reordered ReadAt (parallel readahead, which ReadAt
already supports via RLock + atomics), that exact match frequently misses even for
a genuinely sequential stream, drifting toward random mode and disabling prefetch.
Track a read frontier (max offset+size, advanced with a lock-free CAS loop) and
treat a read as sequential when its start is within SeqTolerance (8 MiB) of the
frontier. The window absorbs reordered/concurrent readahead while still rejecting
far random jumps; the existing ModeChangeLimit hysteresis is unchanged. API
(NewReaderPattern/MonitorReadAt/IsRandomMode) is unchanged, so reader_at is
unaffected. Adds reader_pattern_test.go.
* filer: read the read frontier inside the CAS loop
Capture the frontier from the CAS loop's own load rather than a separate
up-front snapshot, so the sequentiality diff is judged against the freshest
pre-image even if a concurrent readahead advances the frontier while we loop.
Also drops the now-redundant load. Mirrors the same fix on the write-side
ReaderPattern twin (review feedback on the companion PR).
* filer: add frontier/boundary/recovery guard tests for ReaderPattern
Mirror the regression guards added to the write-side WriterPattern: assert
the max-frontier never regresses on a backward read (the CAS invariant), pin
both sides of the inclusive SeqTolerance boundary, and verify sustained near
reads recover sequential mode out of the negative floor. Each guards an
invariant the original four tests left uncovered.
* test: add FUSE database load/durability/perf benchmark
Runs MySQL (InnoDB) and SQLite with ~1GB datadirs on a SeaweedFS FUSE
mount. Two parts:
- durability: normal shutdown, kill -9, and crash-during-write all keep
every fsync-committed row (verified by integrity check + row count +
contiguous prefix + per-row CRC).
- performance: FUSE vs the same local disk. fsync/commit latency is the
dominant cost (~0.13ms -> ~1.18ms), so small transactions run ~9-12x
slower while bulk loads and warm reads stay close.
Harness is path-independent (runtime under $SEAWEED_BENCH_WORK) and only
touches its own processes on non-default ports.
* test/benchmark/fuse_db: portable to Linux + crash-safe progress
- export MYSQL_BIN; mysql_bench.py falls back to PATH when unset
- unmount via fusermount/fusermount3 (non-root Linux), then umount/diskutil
- atomic progress write (tmp+fsync+rename); treat empty progress file as 0
- reuse a single PRNG in the perf probe so RNG init doesn't skew timings
* test/benchmark/fuse_db: validate inputs, add subprocess timeout
- mysql_bench.py: 1800s timeout on mysql CLI calls; reject db names that
aren't plain identifiers (interpolated into SQL)
- sqlite_gen.py / sqlite_verify.py: allowlist journal/synchronous modes and
the verify mode so a typo can't silently weaken durability or relax checks
- run_mysql.sh: durable atomic progress write (tmp+fsync+rename), matching
sqlite_gen.py; quote $LB in both crash-test verify calls
security.toml: document WEED_ env override for the jwt signing keys
These keys are HMAC secrets; spell out the env-var mapping so they can be
injected from a secret store instead of living in the config file.
* fix(command): preserve fuse option after writers
Problem: FUSE mount option parsing skipped the option immediately following concurrentWriters, so values such as concurrentReaders could be silently ignored.
Root cause: runFuse incremented the options loop index inside the concurrentWriters case in addition to the loop increment.
Co-authored-by: Codex <noreply@openai.com>
* test: save and restore mountOptions pointers, not their values
The fields are reassigned to fresh heap variables during runFuse, so
dereferencing to back up/restore mutated throwaways instead of the
flag-bound originals and could nil-panic on unset fields.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* admin: add connected mount clients page and dashboard section
The filer is the authority on who is subscribed to its metadata stream
(FUSE/VFS mounts, S3, peer filers, ...), but its in-memory listener
registry only tracked clientId->epoch and was not exposed.
- Enrich the filer subscriber registry with name/type/address/path/
connected-time, populated in addClient and cleared in deleteClient so
it reflects currently-connected clients only.
- Add a ListMetadataSubscribers filer gRPC (optional client-type filter).
- Admin server fans out to every filer, filters to mount types
("mount" Go weed mount, "sw-vfs" Rust VFS), and renders a new
Cluster > Mount Clients page plus a Mount Clients dashboard section.
Read-only; no behavior change to the subscribe hot path.
* admin: address review — parallelize filer fan-out, guard nil map, robust CSV
- GetMountClients now queries filers concurrently, each under a 5s
timeout, so a slow/unreachable filer can't stall the admin dashboard.
- Defensively initialize fs.subscribers before first write.
- Mount Clients CSV export uses a Blob with quote-escaping instead of a
data: URI, so special characters in paths export correctly.
* filer: keep metadata-subscription send gauge fresh on idle heartbeat
last_send_timestamp_of_subscribe only advanced when a real matching
metadata event was streamed to a subscriber. On a quiet path an idle but
perfectly healthy subscriber therefore looked increasingly stale, and the
dashboard panel rendered a large, misleading 'lag'.
An idle heartbeat is a send too, so advance the gauge when one is emitted.
Subscribers that opt into idle heartbeats (filer.sync) now report true
freshness; the rest still show time since the last real event.
Rename the dashboard panel 'Metadata Subscription Lag' ->
'Time Since Last Subscription Send' and clarify its description to match.
* filer: guard nil option when advancing heartbeat gauge
maybeSendIdleHeartbeat is unit-tested with a bare &FilerServer{} (nil
option), so dereferencing fs.option.Host for the sourceFiler label
panicked. Guard it: production always has option set; the test now gets
an empty sourceFiler label instead of a nil-pointer panic.
* s3: register account-less identities' synthesized account in the lookup
#9962 gave each account-less identity a distinct account id derived from
its name (instead of collapsing into admin), but never registered that
account in the id->account map. GetAccountNameById then returned empty
for such ids, so ACL grantee validation rejected canonical grants to the
caller's own account with InvalidRequest, and bucket/object owner display
was dropped as 'owner is invalid'.
This broke a canned PutObjectAcl by an account-less identity (e.g.
TestVersionedObjectAcl with the default 'some_admin_user' identity):
ValidateAndTransferGrants -> GetAccountNameById -> 'account id is not
exists' -> 400 InvalidRequest.
Register the synthesized account at config load so its id resolves to a
display name. Add a regression test.
* s3: reuse explicitly-configured account for account-less identity
Address review: if an account with the same id as an account-less
identity's synthesized account is explicitly configured (custom display
name/email), reuse it instead of the synthesized one. Add a test.
* metrics: add per-bucket S3 panels and volume slot utilization to dashboard
The S3 Gateway/Buckets rows sliced requests only by type, never by bucket,
though SeaweedFS_s3_request_total and the request/TTFB histograms all carry a
bucket label. Add a Bucket template variable and four per-bucket panels to the
S3 Buckets row (request rate, response codes, request p95, TTFB p95), plus a
Volume Slots Utilization panel (volumes/max_volumes) to the Volume Servers row.
* metrics: scope bucket variable to cluster and guard slot utilization divide-by-zero
Source the Bucket variable from SeaweedFS_s3_bucket_size_bytes (a gauge emitted
for every bucket, including idle ones) scoped to the selected cluster, so the
dropdown lists all buckets in the cluster rather than only those that received
requests. Wrap the volume slot utilization denominator in clamp_min(..., 1) to
avoid +Inf/NaN when max_volumes is briefly zero or absent.
* metrics: guard time()-based panels against unset (zero) gauges
Several dashboard panels compute "time() - <gauge>" (uptime, time-since-
last-scrub, lifecycle cursor lag, last daily walk). When the underlying
gauge is unset it reports 0, so the panel rendered ~56 years (time since
the Unix epoch). This is the common case for "Volume Server Uptime": the
Rust volume server doesn't set SeaweedFS_volumeServer_start_time_seconds,
and Go components expose it registered-but-zero.
Guard each such expression with "> 0" so unset/zero series drop out (the
panel shows no data) instead of rendering a nonsensical epoch value.
Affected panels: Master Uptime, Volume Server Uptime, Time Since Last
Scrub, Lifecycle Cursor Lag, Time Since Last Daily Walk.
* metrics: guard Filer Sync Offset Lag and Metadata Subscription Lag panels too
Extend the > 0 guard to the two remaining time()-<gauge> panels that
share the same unset/zero failure mode (filerSync sync offset and the
metadata subscribe last-send timestamp), so they don't render ~56-year
lag when the gauge is absent.
admin: fold dashboard sparklines into the existing cards
The trend sparklines added in #9957 lived in a separate "Cluster Trends"
row that duplicated the existing summary cards (Volumes, Files, Disk Used,
EC Shards). Remove that row and instead render each sparkline inside the
matching summary card, so every headline number shows its recent trend
without duplication. The two maintenance metrics that have no existing
card — Active Tasks and Workers — now fill the previously-empty columns of
the EC row (also with sparklines).
DashboardTrends changes from a Cards slice to named per-card sparkline
SVGs (+ current values for the two maintenance cards). Drops the now-unused
trendBytes helper (disk size keeps using the existing formatBytes).
* s3tables: resolve account-less identities to a distinct principal
Static identities with no account block default to the shared admin
account, so getAccountID returned "admin" for every such user and the
permission checks treated them all as the admin principal. Only keep the
admin account when the identity actually carries an admin action;
otherwise fall back to the unique identity name.
* s3tables: limit the open-by-default fallback to anonymous access
The legacy permission path allowed any request that no policy explicitly
denied whenever default-allow was on, which is the zero-config default.
That let an authenticated identity without table permissions reach table
resources owned by others. Restrict the fallback to requests with no
identity or the anonymous identity; authenticated callers must pass an
explicit action or policy check. Zero-config and anonymous access are
unchanged.
* s3tables: drop the no-op ListTableBuckets account gate
The top-level check passed the principal as its own owner, so it always
allowed. Per-bucket filtering in the loop is the real authority; remove
the dead gate and the now-unused locals.
* s3tables: derive the Iceberg catalog's default-allow from auth state
The Iceberg catalog reuses the S3 Tables Manager, which hardcoded
default-allow on. Authenticated callers were enforced only because the
identity struct happens to propagate into the handler; if it were ever
dropped, a secured catalog would fall open. Mirror the S3 port and set
the Manager's default-allow from the authenticator, so an authenticated
caller is enforced regardless. Shell and admin keep their own trusted
Manager. Regression test covers the struct, name-only, and admin paths.
* s3tables: drop redundant ACTION_ADMIN string conversion
ACTION_ADMIN is an untyped string constant, so the conversion is a no-op.
* s3tables: enforce name-only authenticated callers, add trusted bypass
defaultAllowFor treated a request with no identity object as anonymous,
but the Manager path forwards only the identity name (not the struct).
A name-only authenticated caller could therefore be misclassified as
anonymous and allowed under the open default. Treat a server-set identity
name as authenticated too, and add an explicit trusted flag for the local
shell/admin tooling that legitimately bypasses authorization.
* s3tables: trim verbose comments
* admin: native at-a-glance trend sparklines on the dashboard
Add a "Cluster Trends" row to the admin Dashboard with inline-SVG
sparklines for volumes, EC shards, disk used, files, active maintenance
tasks, and workers.
The data comes entirely from what the admin already holds — the cached
cluster topology and the in-process maintenance queue — sampled into a
small bounded ring buffer on the existing maintenance-metrics ticker
(~15 min of history). No Prometheus/Grafana dependency, no JS chart
library, no extra goroutine: the sparklines are self-contained SVG
rendered server-side via templ.
This gives basic trend visibility out of the box for clusters that don't
run Prometheus, and a quick glance next to the cluster controls; Grafana
remains the place for deep/historical dashboards.
* admin: cap trendBytes unit index to avoid out-of-bounds panic
A value >= 1 ZiB would push exp past the end of the units string and
panic on units[exp]; cap exp at the last unit (EiB).
* s3: give STS sessions a distinct owner account, not admin
STS sessions were built with Account: &AccountAdmin, so every assumed-role
session shared the admin account for ownership and ACL checks. Use the
assumed-role user as the account id instead, matching the JWT auth path.
Session permissions are unchanged: they come from the session policies,
and admin is granted only through Actions.
* s3: resolve STS session identity to the OIDC subject
Use sessionInfo.Subject (falling back to the assumed-role user when
absent) for the session identity name and account id, so the SigV4 and
JWT auth paths resolve the same session to the same identity instead of
diverging on AssumedRoleUser vs Subject.
* s3: trim verbose comments
* s3: stop collapsing account-less identities into the admin account
Identities configured without an account block all defaulted to the
shared admin account, so distinct users got the same owner id and
ownership checks could not tell them apart. checkAccessByOwnership also
treated that id as an admin bypass, so any account-less caller passed
ownership for any bucket. Give such identities a distinct account id from
their name, and decide the ownership admin bypass by Admin capability
rather than by the account id. isUserAdmin is now nil-safe.
* s3: use the context identity in isUserAdmin before re-authenticating
The Auth middleware already verifies and stores the identity in the
request context. Read it there first so the ownership/admin checks don't
re-run signature verification, which is redundant and fails once the
request body has been consumed.
* s3: nil-guard the context identity in isUserAdmin
A non-nil interface wrapping a typed-nil *Identity passes the type
assertion; guard against it before calling isAdmin().
* s3: trim verbose comments
The bundled dashboard (other/metrics/grafana_seaweedfs.json) covered only
18 of the 84 metrics weed/stats exposes and was a legacy Grafana 8 export
(graph panels, schemaVersion 30). Rebuild it as a modern dashboard
(timeseries panels, schemaVersion 39) with 100% metric coverage, targeting
the direct-scrape model used by Prometheus / seaweed-up / Kubernetes.
- Full coverage of every weed/stats metric: master, volume server, filer,
filer store/sync, s3, s3 buckets, s3 lifecycle, admin/maintenance, build,
wdclient, upload errors, plus Go runtime/process per component.
- Organized into collapsible rows with an always-on Overview.
- Scrape label model: group by `instance`; generic go_*/process_* panels
use `job=~"seaweedfs-.*"` to separate components; an optional `cluster`
template variable (from SeaweedFS_build_info, defaults to All) supports
multi-cluster setups and is transparent when no cluster label is present.
- Same uid (nh02dOVnz) and title so it upgrades in place; drops the dead
"AWS monthly cost" panel.
This is also the single source of truth bundled by seaweed-up's
`cluster dashboard install`.
* refactor(ec_balance): make the balance planner per-volume ratio-capable
Thread a per-volume EC ratio through the balance planner: Plan resolves each
volume's data/parity from a new Options.VolumeRatio (falling back to the
collection Ratio, then the build default, when it reports 0), and keys the
global phase's ratio maps by volume instead of collection. The shell and
worker balance paths build the per-volume lookup from each shard's heartbeat
via the new ecbalancer.VolumeShardRatio.
In OSS this is behavior-preserving: VolumeShardRatio returns 0 because the
per-volume data_shards/parity_shards heartbeat fields are an enterprise
feature, so every volume falls back to the collection ratio -- the existing
standard-scheme behavior. The refactor keeps the shared planner in sync with
the enterprise fork, which overrides VolumeShardRatio to classify and spread
a mixed-ratio collection by each volume's own data/parity split.
* perf(ec_balance): hoist the collection ratio out of the per-volume loop
The collection ratio is constant for every volume in a collection, so
resolve it once per collection instead of per volume; a custom Ratio func
may do map lookups or locking. Addresses a review comment.
* fix(ec): recover EC shards with the volume's own ratio, not the build default
recoverOneRemoteEcShardInterval rebuilt a missing shard with a hardcoded
10+4 Reed-Solomon matrix (and counted sufficiency / iterated shards
against the 10+4 constants). For a custom-ratio volume (e.g. 9+3) that
reconstructs with the wrong matrix and corrupts the recovered bytes, and
cachedLookupEcShardLocations could wrongly reject a degraded but
recoverable custom-ratio read. Use the volume's own ECContext (loaded
from its .vif) for the encoder, the shard-iteration bound, and the
data-shard sufficiency checks. In OSS the ratio is always 10+4 so this is
a no-op; it brings the Go volume server in line with the Rust one, which
already reconstructs with the volume's ratio.
* fix(ec): close data races in the EC read-recovery path
Address review: the freshness check in cachedLookupEcShardLocations read
ecVolume.ShardLocations / ShardLocationsRefreshTime without the lock while
recover goroutines mutate them via forgetShardId -- snapshot both under
ShardLocationsLock.RLock(). The recover goroutines also wrote the shared
is_deleted return concurrently -- collect it via an atomic and fold it in
after they join. Also size availableShards/missingShards by the volume's
ECContext ratio rather than the 10+4 constants.
* fix(ec): remove the shared EC index only when no shard remains node-wide
deleteEcShardIdsForEachLocation removed the shared .ecx/.ecj/.vif index
as soon as a single disk's shard count hit 0, even when a sibling disk
of the same node still held shards of the volume (split-disk reconciled
layout) -- orphaning those shards without their index. Split the
non-teardown delete into two passes: delete the requested shard files
(and now-orphaned per-disk bitrot sidecars) on every disk, then remove
the shared index only once no shard of the volume remains on ANY disk.
This brings the Go volume server in line with the Rust one, which already
gates the index removal on a node-wide check.
* refactor(ec): reuse checkEcVolumeStatus across the two delete passes
Address review: cache hasEcxFile/hasIdxFile from the node-wide count pass
and pass them to removeEcSharedIndexFiles instead of re-listing each
location's directory.
* fix(ec): clean an orphaned EC .vif even when its .ecx is already gone
Address review: removeEcSharedIndexFiles returned early on !hasEcxFile,
so a node-wide teardown left a stale EC .vif behind when its .ecx was
already removed. Decouple the .vif removal (gated on !hasIdxFile) from
.ecx presence so the generation metadata doesn't leak once no shard
remains node-wide.
* fix(storage): delete and unmount every copy of a duplicate volume id
NewStore has no cross-disk duplicate guard (unlike the Rust volume
server, which refuses to start in that state), so a stale twin of a
volume id can mount on a second disk after a disk repair. DeleteVolume
and UnmountVolume returned after the first matching disk, leaving the
twin to survive and re-register as the volume's content. Walk every disk
and act on all copies, emitting one heartbeat delta per copy.
* fix(storage): surface partial delete/unmount failures across duplicate copies
Address review: if removing one copy of a duplicate volume id fails with a
real error (disk IO, permissions), the loop logged it and could still
return success once another copy was removed -- leaving the stale copy to
re-register, the exact divergence this guards against. DeleteVolume and
UnmountVolume now accumulate such errors and return them (still attempting
every disk), so a copy left behind is never reported as success. Add a
DeleteVolume duplicate-copies regression test.
* feat(ec): add encode_ts_ns to the EC task params, shard-unmount, and shard-delete RPCs
The generation fence for stale EC-worker cleanup needs the encode
generation on three messages: ErasureCodingTaskParams (admin issues it),
VolumeEcShardsUnmountRequest, and VolumeEcShardsDeleteRequest (the worker
carries it to the volume server). Additive fields only; 0 preserves the
existing unfenced behavior. Mirror the two volume-server fields in the
Rust volume server's proto copy.
* feat(ec): issue the EC encode generation from the admin and carry it on the worker
Stamp each EC proposal's encode_ts_ns from the admin's per-cycle
DetectionSequence (a single-clock value) so generations are globally
ordered even though detection runs on a rotating worker. The worker
writes that generation into the distributed .vif and passes it on its
shard unmount/delete RPCs; it falls back to a local timestamp for the
.vif only on the unfenced legacy/shell path (keeping the read guard on).
* fix(ec): fence the stale-worker EC shard unmount and teardown by generation
A reaped-but-still-running EC worker's cleanupStaleEcShards issued a
generation-blind unmount + full teardown that could unmount and then
overwrite a newer run's live shards on a shared node. Both RPCs now
carry the encode generation: the volume server unmounts/deletes a disk
only when its .vif generation is strictly older than the request, and
preserves a same-or-newer generation, a generation-0 (recovered or
pre-upgrade) volume, and an unreadable .vif. Unload is per-disk, never
node-wide. Request generation 0 keeps the blanket teardown for the shell
pre-encode cleanup and pre-upgrade callers. Mirrored in the Rust volume
server.
* test(ec): cover the generation-fenced teardown and unmount
End-to-end volume-server tests: a fenced FullTeardown wipes a strictly-
older generation, preserves a newer one, preserves a generation-0 volume,
and blanket-wipes on request generation 0; the gen-aware unmount preserves
a same-or-newer mounted generation; and the .vif generation reader handles
present/absent/no-config cases.
* test(ec): pin the fenced .vif==teardown generation and the unreadable-.vif preserve
A fenced run must stamp the admin generation verbatim into the .vif so it
matches the generation sent on the teardown RPCs; add a regression test
that sets the task generation and asserts the .vif carries it exactly.
Also cover the present-but-unparseable .vif case (reads as generation 0,
preserved) and correct the readEcGenerationTsNs docstring accordingly.
* fix(ec): surface EC full-teardown filesystem errors in the Rust volume server
remove_ec_volume_files(_full_teardown) discarded every fs::remove_file
error, so a teardown that failed on permissions or a full disk still
returned full_teardown_done=true and left stale artifacts to collide with
the next encode. Return io::Result, ignore NotFound, propagate the first
real error, and have the teardown RPC surface it -- matching the Go
contract. The best-effort reconcile/load-cleanup callers keep ignoring it.
* refactor(ec): reuse the EC volume lookup on unmount and short-circuit the gen read
Address review: the Rust unmount fence reuses the ec_vol it already
fetched instead of a second find_ec_volume; the Go .vif generation reader
breaks out of the data/idx loop early when the two dirs are the same.
* s3: preserve checksums for copied multipart parts
* s3: return checksums from multipart copy
* s3: pin the upload's checksum algorithm on copy-part re-stream
* s3: note why UploadPartCopy uses the re-stream slow path
* s3: explain the TLS proxy in the multipart copy checksum test
* s3: cover nil and unknown-algorithm edge cases in copy checksum tests
* s3: cover all checksum algorithms in the multipart copy test
* s3: run all checksum integration tests, not just presigned
* feat(ec): carry the encode generation through the topology heartbeat
Add encode_ts_ns (field 14) to VolumeEcShardInformationMessage and
populate it from each EC volume's .vif identity. The volume server emits
it on the full and incremental heartbeats; the master stores it on
EcVolumeInfo and re-emits it via GetTopologyInfo, so the admin/worker
layer can see which encode run produced each shard set. Field 14 avoids
the enterprise fork's reserved 10-13. Mirror the proto field and both
heartbeat emit sites in the Rust volume server.
* fix(ec): group orphan-source shard completeness by encode generation
countExistingEcShardsForVolume ORed EcIndexBits across every disk, so two
interrupted encode runs whose shard sets overlap unioned into a
false-complete set -- triggering the orphaned-source delete while no
single generation was actually complete. Group shards by encode_ts_ns and
return the largest single generation's count, so the trigger fires only
when one run holds the full set. Shards from pre-upgrade servers
(encode_ts_ns==0) form their own bucket.
The heartbeat carries one encode_ts_ns per (volume, disk), so this
separates generations on different disks; same-disk mixing is prevented
upstream by the pre-encode artifact wipe and the cross-run read guard.
* fix(ec): guard against a nil Ec shard info entry in the generation count
Defensive: a manually-constructed or corrupted topology could carry a nil
entry in EcShardInfos. Skip it rather than dereference.
* fix(ec): carry the encode generation on the EC shard unmount delta
The mount delta sets EncodeTsNs; the unmount deletion delta left it 0.
Populate it from the Ec volume before unloading so both incremental
deltas are consistent (the Rust volume server already does this via its
snapshot diff).
When an EC volume's .vif was missing, NewEcVolume wrote a stub holding
only the version. That stub implies the default 10+4 ratio with
DatFileSize=0 and no encode identity, which the custom-ratio resolver
and the startup credibility checks then read as an authoritative config
-- masking the real ratio of a custom-ratio volume and defeating the
byte-exact .vif gate. Mount with in-memory defaults instead and leave
the real .vif to the encoder or a recovery tool. The Rust volume server
already behaves this way.
* preserve compression state during replication
* explain why ParseUpload skips compression for replica writes
* fix data race on err result in FetchAndWriteNeedle
The local-write and replica-write goroutines all wrote the named err return under an unsynchronized err==nil check. Give each goroutine its own error slot and combine after wg.Wait(): local error wins, then the first replica failure.
* skip redundant decompression of compressed needles during replication
doUploadData decompressed a compressed input only to report the clear-data length on UploadResult.Size, which both replication callers discard. Skip the decompress when IsReplication.
* fix(ec): persist the EC source replica readonly mark
markReplicasReadonly marked each regular replica readonly without
persisting it, so a source-server restart during or after encoding
silently reopened the volume to writes. Those writes are not in the EC
shards, and the later orphan-source cleanup would then delete the
replica, losing them. Send Persist:true so the mark survives a restart;
rollbackReadonly still clears it via VolumeMarkWritable on a failed
encode.
* fix(ec): don't delete a writable source replica during orphan cleanup
cleanupOrphanSourceReplicas issued VolumeDelete to every regular replica
once the EC shard set looked complete, without checking the replica's
current state. A replica that came back writable may hold writes the EC
shards do not contain, so deleting it loses data. Re-probe each replica
via VolumeStatus and skip any that is no longer readonly, logging a
warning instead of deleting.
* fix(ec): check decode .idx writes and fsync decoded .dat/.idx
WriteIdxFileFromEcIndex silently dropped io.Copy and Write errors, so a
short or failed write of the reconstructed .idx went unnoticed and the
caller proceeded to delete the source EC shards. Propagate those errors.
Also fsync the decoded .dat and .idx before returning, so the bytes are
durable before the shards that produced them are removed cluster-wide.
Mirror the .idx fsync into the Rust volume server (its .dat already
syncs and its writes already propagate errors).
* fix(ec): publish decoded .dat/.idx atomically via temp file and rename
WriteDatFile and WriteIdxFileFromEcIndex wrote in place at the final
name with O_TRUNC. A crash mid-write left a truncated .dat/.idx at the
final name beside the still-present EC shards; on restart that partial
file could be mounted as the live volume even though the shards held the
real data. Write to a .tmp file, fsync it, then rename into place and
fsync the directory, so the final name is only ever absent or complete.
A failed decode removes its own temp file rather than leaking it.
Add util.FsyncDir as the shared directory-fsync primitive and reuse the
Rust volume server's fsync_dir for the mirrored change.
* fix(ec): propagate .ecj read errors in the Rust decoder
Path::exists returned false for any error (permission denied, transient
IO), silently skipping the deletion journal and resurrecting deleted
needles as live. Read the journal directly and treat only NotFound as
absent, propagating other errors. The Go decoder already behaves this
way (FileExists returns false only for IsNotExist, then the open
surfaces other errors).
* fix(ec): remove rename destination on Windows in the Rust decoder publish
std::fs::rename does not replace an existing file on every Windows
version. Remove the destination first under a Windows guard before the
atomic publish rename, matching the compaction commit path.
* tiering: stop a shared remote object being deleted while replicas still point at it
A remote-tiered volume's .dat content lives only in one cloud object that all
N replica .vif files point at. Deleting that object while destroying any one
replica, or before a downloaded replica is durable, bricks the survivors.
- volume.tier.move cleanup now deletes old replicas with keepRemoteData=true so
surviving replicas keep the shared object. Document why the alreadyPlaced
anchor needs no replica sync (same-object replicas are byte-identical).
- VolumeTierMoveDatFromRemote now fsyncs the downloaded .dat, fsyncs the
containing directory, trims the .vif (fsynced) and swaps to the local DiskFile
BEFORE deleting the remote object, on both the keep-remote and delete paths.
Only the final DeleteFile is gated by keep_remote_dat_file, so a keep-remote
download leaves the replica served from local disk rather than the shared
object, and a crash before delete merely leaks the object.
- volume.tier.download keeps the shared object for every replica except the
last, which deletes it.
- s3 and rclone download paths fsync the .dat before close.
* storage: swap the volume data backend under the data lock
The tier-download swap closed v.DataBackend and assigned the new local DiskFile
without holding dataFileAccessLock, racing concurrent reads/writes (use of a
closed file / nil deref). Add an exported Volume.SwapDataBackend that performs
the close-and-replace under the lock, and call it from the tier download.
* server: skip directory fsync on Windows in the tier download path
os.Open(dir).Sync() is unsupported on Windows and returns an error, which would
fail VolumeTierMoveDatFromRemote entirely there. Skip the directory fsync on
Windows, matching how the storage-side helper tolerates the unsupported case.
* shell: make multi-replica tier.download resilient to already-local replicas
If a multi-replica download is interrupted and retried, a replica made local
in the prior attempt returns "already on local disk", which aborted the whole
command and left the remaining remote replicas dangling. Treat that case as a
skip-and-continue so a retry completes the rest.
* server: assert downloaded .dat content, not just length, in the tier test
A length-only check passes even if the bytes are corrupted; compare the full
content of the local .dat against the original.
* storage: make vacuum/compaction commit crash-safe with a durable .cpc marker
A crash mid-compaction-commit could lose or corrupt volume data. The
two-rename commit (.cpd->.dat, .cpx->.idx) was not atomic, fsync results
were discarded before renaming over a healthy .dat, a stale .ldb could
poison the needle map, and a duplicate/late commit could delete the live
.dat/.idx outright.
Introduce a durable .cpc commit marker so the swap is atomic across a
crash:
- CommitCompact writes and fsyncs the .cpc marker after makeupDiff
fsyncs the .cpd/.cpx, then runs applyCompactSwap: an existence-guarded
rename of .cpd->.dat and .cpx->.idx, a directory fsync, removal of the
stale .ldb/.rdb, and finally removal of the marker.
- reconcileCompactState recovers an interrupted commit on load: roll
forward (finish the renames) when the marker is present, roll back
(delete the orphan .cpd/.cpx) when it is absent. It runs from a
directory pre-pass keyed on .cpd/.cpc existence, since the per-volume
loader is keyed on .idx/.vif and misses the marker-only and
already-renamed-.idx states.
- applyCompactSwap verifies BOTH .cpd and .cpx exist before touching the
live files, so a stale-state commit (including the Windows
RemoveAll-then-rename path) errors without deleting anything.
- Error-check the fsyncs that gate the swap: the .cpd close-fsync and
.cpx fsync in copyDataBasedOnIndexFile, the makeupDiff .idx fsync, and
MemDb.SaveToIdx.
- generateLevelDbFile rebuilds from offset 0 when the stored watermark
sits past the end of the .idx, instead of replaying zero entries and
poisoning the needle map.
- removeVolumeFiles and cleanupCompact sweep the .cpc marker; cleanup
refuses to unlink the temp files while a marker is present.
Mirror the commit-marker, fsync-before-rename, guard, and
load/reconcile logic in the Rust volume server.
* storage: don't reconcile an already-loaded volume's compaction state on reload
reconcileCompactStates runs in loadExistingVolumes, which is re-invoked at
runtime on SIGHUP (Store.LoadNewVolumes). For a volume that is already loaded
and mid-vacuum, its .cpd/.cpx are live temp files, not crash leftovers --
rolling them back would clobber the in-flight compaction (and remove a live
.ldb out from under an open handle). Skip any vid already present in the
volume map; genuine startup recovery runs before any volume is loaded, so the
map is empty then. Mirrored in the Rust volume server.
Also drop the .note keepVif change that crept into this branch; it belongs to
the replica-copy/verify workstream and is restored to master's behavior here
so the two changes don't collide.
* storage: roll a compaction commit forward per-file, not all-or-nothing
A crash after the .cpd->.dat rename but before .cpx->.idx leaves .cpd gone,
.cpx and .cpc present, and a stale .idx. The roll-forward required BOTH temp
files, so it skipped the swap and cleared the marker, pairing the fresh .dat
with the stale .idx (index corruption). Finish whichever temp file remains:
extract finishCompactSwap to rename .cpd->.dat and/or .cpx->.idx independently;
applyCompactSwap keeps the both-present guard for the normal commit. Existence
in the Rust mirror is checked robustly so a transient error never skips the swap.
* seaweed-volume: propagate directory fsync failures on the compaction commit path
fsync_dir dropped every sync_all error, so the commit could proceed with an
undurable marker or rename and a later restart could recover the wrong
generation. Return the error and check it at the commit call sites (marker write
and the swap), matching the Go fsyncDir which already propagates. Directory
fsync stays a no-op on Windows, where it is unsupported.
* storage: overflow-safe stale-watermark check when rebuilding the leveldb index
watermark*NeedleMapEntrySize can overflow uint64 for a corrupted watermark and
wrap below the file size, defeating the stale-.ldb guard. Compare in entries
(watermark > size/NeedleMapEntrySize) instead, which is equivalent and cannot
overflow. LevelDb-backed needle map is Go-only; no Rust mirror.
* storage: propagate idxFile.Close error when writing the compacted index
SaveToIdx writes the .cpx that is renamed to .idx at commit; a discarded Close
error (buffered data not flushed) could leave a partially-written index after a
crash. Surface it in the same durability gate as the fsync.
* volume: verify before destroy in VolumeCopy and replication repair
Four data-safety fixes around copy/repair paths that could destroy or
resurrect data before verifying the source or survivors.
(a) VolumeCopy no longer deletes a pre-existing local replica up front.
The delete is deferred until ReadVolumeFileStatus on the source succeeds,
so a transient source outage (or a retry after one) can no longer wipe a
healthy destination replica. Gated on source readability only; size/count
comparisons are intentionally not used because they invert legitimately
after divergent vacuum/compaction. Mirrored in the Rust volume server.
(b) volume.check.disk no longer resurrects vacuumed-deleted needles. A
key present-and-live on the source but entirely absent on the target is
ambiguous: it may be a genuine missing write, or a needle deleted on the
target and then vacuumed (its index entry and any tombstone are gone). An
individual needle AppendAtNs has no monotonic relation to a vacuum
watermark, so the old cutoff heuristic could not tell them apart. Without
positive proof the absence is a missing write, the safe default is to NOT
push it back. Tradeoff: a real missing write may go unrepaired until a
tombstone-aware path exists, but we never raise back deleted data.
(c) Over-replication trim no longer resurrects needles or removes the
wrong replica. The pre-delete sync now runs read-only (divergence check
only) instead of writing the doomed replica's needles into the survivor.
pickOneReplicaToDelete only ever removes the smallest of multiple healthy
writable replicas; it refuses the trim when doing so would leave only
read-only/integrity-flagged survivors, since file_count>0 alone cannot
prove the survivor's .dat is readable.
(d) Incomplete-volume (.note) cleanup keeps the shared .vif when an .ecx
for the same vid coexists on the disk, so removing an interrupted regular
copy cannot strip a coexisting EC volume's info file. VolumeCopy now
surfaces .note write/remove errors instead of ignoring them. In the Rust
volume server (where a persisting note is actually reachable) the .note
check moves below the empty-stub sweep and EC validation, keeps the .vif
on EC coexistence, and the mount path fails when a .note still persists.
* shell: scope the over-replication writable-survivor guard to the trim path only
The writable-survivor guard (never trim down to a read-only survivor) lived
inside the shared pickOneReplicaToDelete, so it also gated the misplaced-volume
relocation via pickOneMisplacedVolume -- a misplaced read-only volume (e.g. a
full one) would silently stop being rebalanced. Extract pickSmallestReplica
for the relocation path (which deletes-and-recreates and must act on read-only
replicas), and keep the writable-survivor guard only in pickOneReplicaToDelete
used by the over-replication trim.
* seaweed-volume: recompute keep_vif after invalid-EC cleanup in the .note path
keep_vif used the pre-validation ecx_exists snapshot, so when the EC-validation
step above removed the invalid .ecx/shards, the .note cleanup still preserved a
now-orphaned .vif. Re-check .ecx existence at cleanup time, matching the Go
hasEcxFile re-check.
* shell: keep placement when picking an over-replication victim to delete
The trim picked the smallest writable replica without regard to placement, so
it could delete the only replica in a required failure domain (e.g. with "100"
and replicas dc1 + two in dc2, deleting dc1 leaves both survivors in dc2).
Prefer a writable replica whose removal still satisfies placement, falling back
to the smallest writable only when none does.
* fix(tier): preserve volume data modification time
* fix(tier): best-effort restore of data mtime on download
A failed Chtimes should not abort an otherwise complete tier-down; warn
and continue, matching the EC copy path.
* fix(tier): preserve volume data mtime in rust volume server
Mirror the Go fix: store the source .dat mtime on upload instead of the
upload time, and restore it on the downloaded .dat. Without this a
tiered-then-restored volume loads last_modified_ts_seconds from the
upload/download time, extending its TTL across a restart or remount.
* fix(tier): read source mtime via DiskFile.GetStat()
GetStat() is nil-safe when the backend is closed concurrently and skips a
redundant stat syscall; its cached modTime is the on-disk mtime a reload
reads, since every .dat write or Chtimes is followed by a DiskFile (re)open.
* fix(tier): surface mtime-restore failures on rust tier-down
set_file_mtime now returns io::Result; the tier-down path warns on a
failed restore instead of dropping it silently, so a wrong local .dat
mtime (and the TTL drift it causes) is observable. Matches the Go
download. The EC copy path keeps its best-effort silence.
* fix(ec): never delete recoverable shards on startup/reconcile (size-direction + byte-exact .dat)
EC startup validation and the cross-disk reconcile could delete the only
copy of distributed-EC shards whenever a non-empty .dat sat beside them.
This is the same data-loss class as the empty-.dat-stub fix, now for a
real (non-empty) stale or partial .dat.
validateEcVolume: the discriminating signal is the shard size relative to
the .dat's full encode, not the shard count.
- shards smaller than expected: an interrupted local encode left partial
shards and the .dat is the complete source -> reclaim the .dat.
- shards equal to expected: a valid (or still-distributing) EC volume ->
keep; the shards may be the only copy.
- shards larger than expected: the .dat is the stale/partial side (e.g. an
interrupted decode left a half-written .dat next to the real shards) ->
keep.
Previously any size mismatch, a low shard count beside a .dat, or a
transient stat error returned "delete", wiping sole-copy shards. Now every
ambiguity (size mismatch in either direction, inconsistent shard sizes,
transient I/O error, partial shard set) keeps the data; only a credible
full source .dat with no partial set to lose is reclaimed.
handleFoundEcxFile: a shard load failure (corrupt/locked .ecx, EMFILE
during a mass restart, transient I/O) no longer deletes the EC files when a
.dat exists -- it only unloads and keeps the files for retry. All deletion
authority now flows through validateEcVolume.
pruneIncompleteEcWithSiblingDat: count shards NODE-WIDE (a set split across
sibling disks summing to >= dataShards is independently recoverable and is
left alone), and require the sibling .dat to byte-exactly match the size
.vif recorded at encode time before deleting -- the prior "at least this
big, or bigger than a superblock" gate could trust a stale .dat and wipe
sole-copy shards. EC encode records the source size in .vif, so this gate
works for real volumes; older volumes without it fail safe (kept).
Rust volume server mirrors all of the above: size-direction + keep-on-
ambiguity in validate_ec_volume, keep-on-load-failure in
handle_found_ecx_file, and the node-wide + byte-exact gate in the prune.
The Rust validate/prune paths now resolve the data-shard count from the
volume's own .vif instead of hardcoding 10+4, so custom-ratio volumes are
not mis-sized and wrongly deleted on reboot.
Existing tests that encoded the old (unsafe) "delete on low count / size
mismatch" behavior are updated to the safe expectation, and new regression
tests cover the partial-decode-.dat-keeps-shards and transient-error-keeps
cases (Go and Rust); they fail on the pre-fix code.
* fix(ec): record DatFileSize in planted EC .vif for the prune test; trim comments
The multi-disk lifecycle e2e test planted a partial EC leftover with an
empty .vif, so the byte-exact prune gate (which a real encoded volume
satisfies via its recorded source size) kept it instead of cleaning up.
Record DatFileSize + the EC ratio in the planted .vif, matching production.
Also condense the verbose comments added in this change to the repo's
concise style.
* shell: stop ec.encode/ec.rebuild from destroying live EC shards
Three operator-triggered shell paths could destroy data with no crash:
ec.encode -volumeId on an already-EC volume tore down its shards before
failing. The volume-id path never checked the id was a regular volume:
the collection lookup scans only VolumeInfos (so an EC-only id maps to
""), and volumeLocations succeeds via the EC-location fallback, so
clearPreexistingEcShards full-teardown-deleted every shard cluster-wide
before doEcEncode failed. An EC volume has no .dat, so this is its only
copy. Add assertEncodableRegularVolumes: each requested id must be a
regular volume in the topology snapshot; an EC-only or unknown id is
refused before any teardown. A volume present as both a regular .dat and
stale orphan shards (a failed-encode retry) still passes. This closes
the operator-rerun/script-retry path; a worker racing the snapshot is a
fencing problem handled separately.
ec.rebuild dry-run (the default, without -apply) still issued real
VolumeEcShardsDelete RPCs: prepareDataToRecover appended every
would-copy shard to copiedShardIds even though the copy was skipped, and
the cleanup defer deleted that set unconditionally. Now a dry-run copies
nothing and records nothing to delete (a separate would-copy counter
drives the recoverability check so the dry-run still reports its plan),
and the cleanup runs only under -apply.
ec.rebuild could also self-destruct a live shard: localShardsInfo was
overwritten per disk instead of unioned, so a shard the rebuilder holds
on a non-last disk looked remote, got copied onto itself (in-place
O_TRUNC) and then node-wide deleted. Union local shards across all
disks, and never copy/delete a shard whose only listed holder is the
rebuilder itself.
* shell: address ec destructive-guards review comments
- countLocalShards: union shards across all of the rebuilder's disks so
slot accounting matches what prepareDataToRecover treats as local;
first-match counting overstated slotsNeeded on multi-disk rebuilders
- VolumeEcShardsCopy: resolve SourceDataNode via
pb.NewServerAddressFromDataNode instead of the raw node id, which may
not be a dialable host:port
- assertEncodableRegularVolumes: skip nil DiskInfo map entries, matching
the other topology walks in this file; rename ecOnly to hasEcShards
since the map marks any volume with shards, not only shard-only ones
* fix(ec): make ecx-journal fold and shard rebuild crash-safe
Two EC rebuild paths could silently lose or corrupt data:
RebuildEcxFile folded the .ecj deletion journal into .ecx (in-place
WriteAt tombstones) and then unlinked the journal without flushing the
.ecx writes first. A crash could persist the unlink ahead of the
tombstones, resurrecting deleted needles on the next load. It also read
journal records with a bare n!=size break, so a torn tail silently
dropped the remaining tombstones before the unlink. Now: read records
with io.ReadFull (io.EOF ends cleanly, a torn tail aborts and leaves
.ecj in place for retry), fsync .ecx before removing the journal.
rebuildEcFiles treated a zero/short ReadAt as a clean end-of-input and
discarded the read error, so a truncated or unreadable input shard
produced truncated regenerated shards that were then published as
restored redundancy; the regenerated shards were also never fsynced on
the no-sidecar path. Now: derive the expected shard size from the
present inputs up front (rejecting a divergent/zero-size input), drive
the loop by that size, fail on any short read or short write, and fsync
every regenerated shard before it is mounted/renamed.
Rust volume server mirrors the rebuild fix: rebuild_ec_files now checks
the read_at byte count (it previously discarded it, the same truncation
bug). The Rust ecx fold already synced .ecx before removing the journal.
Custom EC ratios are unaffected: the shard size derives from the input
shards and the loop uses the .vif-resolved data/parity counts, never a
hardcoded 10+4.
* storage: close ecx journal files via defer in RebuildEcxFile
Per review: a single deferred Close per file replaces the per-error-path
manual closes, so new early returns cannot leak descriptors. The journal
is still closed explicitly before its unlink since Windows cannot delete
an open file; the deferred second Close is a harmless no-op.
* fix(util): ignore comment only sql input
Problem: sqlutil.SplitStatements strips SQL comments while scanning, but when no statements remain it falls back to returning the original query. Inputs that contain only comments are therefore reported as executable SQL statements.
Root cause: The no-statements fallback did not distinguish a real single statement from input that had been fully removed by comment filtering.
Fix: Remove the original-query fallback and return an explicit empty slice when scanning produces no statements.
Reproduction: env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/util/sqlutil -run TestSplitStatements -count=1 failed before the fix because comment-only inputs returned the comment text as a statement.
Validation: gofmt -w weed/util/sqlutil/splitter.go weed/util/sqlutil/splitter_test.go; env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/util/sqlutil -run TestSplitStatements -count=1; env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/util/sqlutil -count=1; git diff --check; git diff --cached --check.
Duplicate check: Searched /private/tmp/seaweedfs-codex0610-old-branch-index.tsv and existing tests for sqlutil, SplitStatements, comments, and comment-only. Old PostgreSQL query branches cover malformed wire frames and SQL engine numeric parsing, not comment-only statement splitting.
Co-authored-by: Codex <noreply@openai.com>
* Update weed/util/sqlutil/splitter.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* s3: validate indirect filer path inputs
* s3: avoid query parsing on common request path
* filer: scope copy/move source against JWT AllowedPrefixes
maybeCheckJwtAuthorization only checked r.URL.Path, but copy and move read
their source from the cp.from / mv.from query params. A prefix-restricted
token could copy or move data out of a subtree it cannot otherwise reach.
Check every path the request touches, reusing pathHasComponentPrefix so
`..` in the source is collapsed before the prefix match.
* s3: confine iceberg CreateTable location to the catalog bucket
CreateTable derived the metadata bucket and path from the client-supplied
req.Location / req.Name and wrote there directly, so a caller scoped to one
table bucket could place metadata in another bucket (and path.Join collapsed
any `..`). Require the parsed bucket to equal the request's catalog bucket
and reject traversal segments in the table path.
* webdav: clean client path before subFolder confinement
wrappedFs concatenated subFolder + name before the underlying FileSystem
ran path.Clean, so `..` in the request path or COPY/MOVE Destination
resolved across the FilerRootPath confinement boundary. Clean the name as a
rooted path first so traversal segments collapse below subFolder. Only the
non-default -filer.path (non-empty subFolder) setup was affected.
* filer: enforce read-only rule on real write path with destination header
The x-seaweedfs-destination header overrides the path used for storage-rule
matching while the entry is written at r.URL.Path, letting a caller select a
writable rule for a read-only target. When the header is present, also check
the read-only/quota rule against the actual write path.
* fix(storage): never let an empty .dat delete healthy distributed EC shards
A leftover empty .dat stub (a phantom from the pre-fix loader; zero
needles) next to a distributed EC volume's local shards made startup
classify the volume as an interrupted local encode: validateEcVolume
requires >= dataShards local shards when a .dat is present, fails with
the 1-2 shards a distributed volume keeps per disk, and the cleanup
deletes those shards -- the only copies of that part of the volume.
Repeated across restart waves this destroys enough shards cluster-wide
to make the volume unrecoverable.
Go:
- loadExistingVolume: hoist the empty-stub sweep above the EC presence
checks. Previously the .vif-next-to-.ecx guard returned before the
sweep ever ran, so exactly the dangerous layout (stub + .ecx + local
shards) kept its stub and then lost its shards in loadAllEcShards.
- validateEcVolume / checkDatFileExists: treat a .dat <= a superblock
(zero needles) as absent. An empty .dat cannot be the encode source,
so it must never gate shard deletion; this also covers stubs without
a .vif, which the sweep cannot prove are EC leftovers.
Rust mirror (seaweed-volume): the same gate in validate_ec_volume and
check_dat_file_exists (the Rust sweep already ran before validation);
the volume-load skip keeps a plain existence check so fresh,
needle-less volumes still load.
Regression tests in Go and Rust reproduce the production layout (a
zero-byte .dat beside .ecx/.ecj and two shards of a 10+4 volume, with
and without a .vif) and fail without the fix with the shards deleted.
* fix(ec): gate source volume deletion on a recoverable shard set
After EC encode, the shell command and the (plugin) worker task refused
to delete the source volume unless every shard was present, and aborted
otherwise -- leaving the source .dat next to live shards, exactly the
mixed state the startup cleanup mishandles.
Replace the full-set requirement with a recoverability gate shared by
both callers (RequireRecoverableShardSet): deleting a non-empty source
.dat requires at least dataShards distinct shards cluster-wide. Below
that the source is kept and the encode fails as before. A degraded but
recoverable set (>= dataShards, < total) now proceeds with a warning
instead of aborting: the missing shards can be rebuilt from the
survivors, while keeping the source would preserve the dangerous mixed
state. Empty stub replicas are still swept unguarded (OnlyEmpty) -- an
empty .dat has nothing to lose.
dataShards/totalShards stay parameters so enterprise custom EC ratios
share the helper verbatim.
* test(ec): use recoverable shard verification gate
Reject copy sources whose bucket/object fail IsValidBucketName /
IsValidObjectKey, the helpers validateRequestPath already applies to the
request URL. The object is joined onto the bucket path and `.`/`..`
segments are collapsed by the filer, so without this the source need not
stay within the parsed bucket. Route UploadPartCopy through
ValidateCopySource too; it previously only checked for empty bucket/object.
* feat(storage): sweep leftover empty EC .dat stubs on volume server startup
An EC volume keeps no local .dat. The pre-fix loader left empty 8-byte
superblock .dat stubs next to EC metadata (one per lone .vif). Left in
place each loads as a phantom empty volume, and the same vid's stub on
two disks of one server blocks Rust startup via the duplicate-vid check
in Store::add_location -- the prior fix stops creating new stubs but does
not clean up existing ones.
On startup, when a .dat is empty (<= a superblock, i.e. zero needles) and
its .vif marks the volume erasure-coded, remove the stub (+ empty .idx)
instead of loading it. The real data is in the EC shards, so the empty
stub holds nothing to lose. Non-EC empty .dat files (e.g. freshly
allocated volumes) are left alone.
Done in both Rust (load_existing_volumes) and Go (loadExistingVolume),
with regression tests that fail without the sweep.
* refactor(storage): extract empty EC .dat stub sweep into its own function
Move the startup stub-sweep into remove_empty_ec_dat_stub (Rust) and
removeEmptyEcDatStub + vifIsEcVolume (Go) for clearer logic, and look up
the .vif in both the data and idx directories (each read at most once) so
a stub is still found when -dir.idx is configured. Adds direct tests for
the idx-directory lookup on both engines.
* admin: configure maintenance tasks via admin.toml
Maintenance task settings could only be edited in the admin UI and live
under <dataDir>/conf, so they silently reverted to defaults whenever the
data directory was recreated. An optional admin.toml now declares vacuum,
balance, and erasure coding settings; keys set there are written through
to the persisted task configs at every startup, overriding UI edits, so
the configuration stays declarative. Generate an example with
"weed scaffold -config=admin".
* vacuum: round min volume age up to whole hours
MinVolumeAgeSeconds was truncated by integer division when converted to
the hour-granular protobuf field, so a sub-hour setting silently became
0 and disabled the age guard.
* admin: split and normalize preferred_tags from admin.toml
A comma-separated string, as set via environment variable, came through
viper as a single slice element. Split on commas and reuse
util.NormalizeTagList, matching the plugin config path.
* scaffold: clarify admin.toml wording
* shell: volume.tier.move can move volumes between data centers
-fromDataCenter scopes volume selection to volumes with a replica in
that data center. -toDataCenter constrains move destinations and
replication fulfillment. With identical disk types both flags are
required, moving full volumes between data centers on the same tier.
* shell: assert node identity in data center filter test
* shell: tier move resumes when the volume is already on the target
A replica already on the target tier and data center, typically left by
an interrupted earlier run, anchors the move: skip the copy and only
complete replication fulfillment and old replica cleanup. Previously
such volumes hit the no-destination path and the stale source replicas
were never removed.
386 test binaries execute natively on the amd64 runner, so the suite
catches what vet cannot: unaligned 64-bit atomics and arithmetic that
wraps at runtime. -short keeps the e2e suites on amd64 only.
* fix(tests): keep EC e2e fid cookie arithmetic in uint32
The cookie constants 0x9490CA00 and 0x9500CA00 were added to the int
loop variable before conversion, overflowing 32-bit int at compile
time on linux/386 and linux/arm. Convert the loop variable instead so
the addition stays in uint32.
* fix(tests): pass s3client max backoff in milliseconds
MaxBackoffDelay is documented as milliseconds and multiplied by 1e6
before use, but the example set it to 5s in nanoseconds, yielding an
absurd backoff on 64-bit and a compile-time int overflow on 32-bit.
* ci: type-check code and tests for linux/386
64-bit-only constant arithmetic keeps slipping into test files and
breaking 32-bit downstream builds. Vet the whole root module under
GOOS=linux GOARCH=386 so these fail in CI instead of after release.
* fix(tests): convert s3client backoff to Duration before scaling
The ms-to-ns multiplication ran in int, wrapping at runtime on 32-bit;
scale by time.Millisecond after the Duration conversion instead.
The quota enforcement loop already computes each bucket's configured
quota and effective read-only flag every minute, but neither was
visible to monitoring, so operators could not alert before a bucket
flips read-only.
Add two gauges next to the existing bucket size metrics:
SeaweedFS_s3_bucket_quota_bytes configured quota; the series is only
present while the quota is enabled,
so size/quota utilization queries
never divide by zero
SeaweedFS_s3_bucket_read_only 1 when the bucket's location rule is
read-only (over quota or manually
locked), 0 otherwise
Both are cleaned up with the other per-bucket gauges on bucket
deletion and inactivity TTL.
* feat(ec): add encode_ts_ns to EC shard metadata and the shard read RPC
EcShardConfig and VolumeEcShardReadRequest gain an int64 encode_ts_ns
(encode time in unix nanos). It rides in .vif and the read request so a
read can be scoped to the encode run that produced the index.
* fix(ec): stamp each encode and reject cross-run shard reads
Generate stamps EncodeTsNs into the volume's .vif. Reads carry it to the
shard's owning volume (resolved together via FindEcVolumeWithShard, so a
multi-disk server validates the disk that actually serves the bytes) and
reject a shard from a different encode run, recovering from parity. A
zero on either side (pre-upgrade volume) skips the guard.
* fix(ec): stamp the encode identity on the worker-generated .vif
The worker-local encode path now writes EncodeTsNs (and the resolved EC
ratio) into the .vif, so the read guard is not silently off for volumes
encoded by the maintenance worker.
* fix(ec): wipe stale EC artifacts before re-encoding
VolumeEcShardsGenerate evicts any in-memory EcVolume for the volume and
removes its on-disk shard/index/sidecar files before writing fresh ones,
so a retried encode never builds on a partial prior run and the unlink
frees the inodes instead of leaving open fds serving old bytes.
* fix(ec): unmount EC shards across all disks
UnmountEcShards walked only the first disk holding the shard, leaving a
duplicate copy mounted on a sibling disk (split-disk reconciled volumes)
still serving and heartbeating. Traverse every disk and emit one
deletion delta per disk.
* fix(ec): delete orphan shards without a local .ecx
deleteEcShardIdsForEachLocation gated shard-file removal on a local .ecx,
so it could not clean an orphan .ecNN left by a failed copy on a disk
with no index. Delete the requested shard files unconditionally; the
index-file (.ecx/.ecj/.vif) routing stays gated as before.
* fix(ec): clear stale EC shards cluster-wide before re-encoding
ec.encode unmounts and deletes EC shards for the target volumes on every
node before regenerating: fatal for the shards the topology reports
(mounted leftovers), best-effort for the rest (a sweep that catches
unmounted failed-copy orphans). A down node is a no-op.
* fix(ec): don't nil EC fds on close so reads can't race eviction
A reader resolves an EcVolume/shard under the lock then reads after it is
released, so an eviction that nils ecxFile/ecdFile would race that read
and panic. Close the fds without nilling the fields: the field is now
write-once (no data race) and a concurrent read hits a closed fd, getting
a clean error that the caller recovers from parity.
* fix(ec): wipe stale EC artifacts on every disk and surface failures
The pre-encode wipe only deleted beside the source volume, so a stale
shard on a sibling disk survived and could be mounted against the new
index at reconcile. Sweep every disk. Removal also ignored os.Remove
errors, reporting a failed cleanup as success and letting a stale shard
join the next generation; surface the first real failure (treating
already-gone as success) from removeStaleEcArtifacts and the shard delete.
* fix(ec): log when a local shard is skipped for a different encode run
The cross-run guard returned errShardNotLocal, indistinguishable in logs
from a genuinely-absent shard. Add a V(1) line naming both EncodeTsNs so
operators can tell "wrong encode generation" from "shard not here".
* fix(ec): surface metadata removal failures in the shard delete path
deleteEcShardIdsForEachLocation still dropped os.Remove errors on the
.ecx/.ecj/.vif/sidecar cleanup. A surviving stale .ecx is the orphan-index
condition this path prevents, so route those through removeFileIfExists and
return the first real failure instead of reporting cleanup as success.
* fix(ec): fail orphan cleanup when a reachable node's delete fails
The pre-encode orphan sweep swallowed every error for unreported (node,
volume) pairs. That is only safe for an unreachable node, which cannot
receive this encode's new generation. A reachable node whose delete
genuinely failed (permission/IO) keeps an orphan shard that a later copy
re-stamps with the new run's volume-level .vif identity, so the read guard
would accept stale data. Surface those; stay best-effort only for
unreachable nodes (gRPC Unavailable / no status).
* fix(ec): guard ecjFile under its lock in the EC delete path
EcVolume.Close nils ecjFile under ecjFileAccessLock; a delete that resolved
its .ecx lookup before a concurrent eviction (the generate-time
UnloadEcVolume) could then reach the journal append with a nil fd. Bail
with a clear "volume closed" error under the lock instead.
* fix(ec): reject an unstamped shard when the caller has an encode identity
The read guard required both identities nonzero, so a current (stamped)
caller accepted a holder with identity 0 and could be served a stale
pre-upgrade shard. Reject when the caller is stamped and the holder
differs (including unstamped); stay lenient only when the caller itself
has no identity (pre-upgrade reader). A skipped shard recovers from parity.
* fix(ec): full-teardown delete so cluster cleanup wipes a whole generation
The pre-encode cluster sweep deleted only the listed canonical shards on
remote nodes, leaving index/sidecar (and, on builds with versioned
generations, those too) behind. Add a full_teardown flag to
VolumeEcShardsDelete that evicts the volume and wipes every EC artifact for
it on every disk via removeStaleEcArtifacts; the shell and worker pre-encode
cleanup paths set it. Other delete callers (balance/decode/repair) are
unchanged.
* fix(ec): take ecjFileAccessLock before the nil-check in Sync and Close
Sync and Close read ev.ecjFile before acquiring ecjFileAccessLock while
Close nils it under the lock, a data race on the field. Take the lock
first, then nil-check inside, in both.
* fix(ec): acknowledge full_teardown so a pre-upgrade server can't fake success
An old volume server silently ignores full_teardown and returns success
for an ordinary delete, so the caller wrongly believes the generation was
wiped and copies a fresh gen-0 onto an unwiped node. Echo full_teardown_done
in the response; the worker destination cleanup fails when it is absent, and
the shell cluster sweep fails for a reported (mounted) leftover while staying
best-effort for an unreported node. encode_ts_ns stays an accepted transient
(an old server just skips the new read guard, no regression).
* fix(ec): fail the pre-encode sweep for any reachable node that can't ack teardown
A reachable pre-upgrade server ignores full_teardown and returns success
without wiping an orphan, which a later copy then folds into the new
generation. Treat a missing full_teardown_done ack as fatal for every
reachable node (best-effort only for a gRPC-unreachable one), not just for
topology-reported pairs.
* fix(ec): return the served shard identity and validate it client-side
The encode identity was only enforced server-side, so a pre-upgrade server
ignored the request field and served bytes unchecked. Echo the served
shard's EncodeTsNs on every read response chunk and have the client reject a
mismatch (including 0 from an old server), so the guard holds regardless of
server version; a rejected read recovers from parity.
* fix(ec): reject a short/empty remote shard read instead of serving zeros
doReadRemoteEcShardInterval accepted an immediate EOF or a short stream and
returned success with a partly zero-filled, unvalidated buffer (the server
stamps the identity only on chunks that carry bytes). A non-deleted interval
must arrive whole: require n == len(buf), exempting the is_deleted
short-circuit (n=0), matching readLocalEcShardInterval's local check. A short
read now fails so the caller recovers from parity.
* test(ec): fake volume server echoes the full_teardown acknowledgement
The worker now fails a teardown delete that isn't acknowledged (so a
pre-upgrade server can't silently skip the wipe). The fake server's no-op
VolumeEcShardsDelete returned an empty response, which the worker read as a
skipped teardown and aborted the encode. Echo full_teardown_done.
* feat(ec): mirror the encode-run identity guard + full_teardown into the Rust volume server
The Go volume server stamps an encode-run identity (encode_ts_ns) into the .vif
and rejects a read served from a shard of a different run; full_teardown wipes a
whole generation and acknowledges it. The Rust volume server had none of it.
Mirror the shared logic: load encode_ts_ns from the .vif onto the EcVolume,
stamp it on every read response, and reject a request/response mismatch on both
the server and the distributed-read client (recovering from parity); handle
full_teardown by evicting the volume and wiping every EC artifact on each disk,
echoing full_teardown_done so the caller can detect a server that ignored it.
* fix(ec): remove a stale .vif on full teardown of a shard-only node
A shard copy installs shards + .ecx before .vif, so an interrupted copy after a
teardown could mount the new files under the previous run's identity / version /
shard ratio / dat_file_size carried by the surviving .vif. Remove .vif during
full teardown, gated on .idx absence so a source-volume holder keeps its live
.vif. In Rust this lives in a teardown-only helper so the reconcile / load-
fallback paths (which share the base removal) still preserve .vif.
* fix(ec): treat a missing teardown ack as fatal, not as an unreachable node
isNodeUnreachable returned true for any non-gRPC-status error, so a reachable
pre-upgrade server's missing full_teardown_done ack (a plain error) was
classified unreachable and the unreported pair was silently skipped. Classify
only a real codes.Unavailable as unreachable, and wrap the missing ack in a
sentinel the sweep treats as fatal regardless. A genuinely down node still
surfaces as Unavailable from the RPC and stays best-effort.
* fix(ec): reject a short shard read in the local EC needle reader
read_ec_shard_needle ignored the byte count from shard.read_at and appended the
whole pre-sized buffer, so a truncated shard's zero-filled tail passed the later
length check and parsed as garbage. Require n == buf.len() per interval, erroring
on a short read like the local interval reader already does.
* fix(ec): probe reachability before skipping a node that returns Unavailable
The pre-encode sweep skipped any node whose teardown delete returned
codes.Unavailable, but a reachable volume server in maintenance mode also
returns that code for the maintenance-gated delete, so its stale EC files were
left behind on a node that can still receive the new generation. Confirm with a
non-maintenance-gated empty-target Ping: skip only when the node fails the probe
too (genuinely unreachable).
* fix(ec): use try_exists for the teardown .vif .idx guard
The teardown-only .vif removal gated on Path::exists(), which returns false on a
permission/IO stat error, so a stat failure on a present .idx would read as a
shard-only node and delete the live source volume's .vif. Gate on
try_exists() == Ok(false) instead, preserving the sidecar on any stat error.
* fix(ec): only skip a sweep node when a Ping confirms it is transport-down
The pre-encode sweep skipped a node whenever its teardown delete and a liveness
Ping both failed, but it treated ANY Ping error as down — an application-level
Internal/ResourceExhausted, or Unimplemented from a pre-Ping server, left a
reachable node's stale generation in place. Classify the Ping tri-state and skip
only when it transport-fails with codes.Unavailable; a reachable or inconclusive
node stays fatal.
* fix(ec): exclude sweep-skipped nodes from the encode's rebalance
The pre-encode sweep skips a genuinely-down node best-effort, but the rebalance
then recollected the current topology — a node that recovered between the two
could become a copy target and receive the new generation while still holding
its stale, never-cleared shards. Have the sweep return the skipped set and
exclude those nodes from the rebalance for this encode, so a node we could not
clean cannot receive the new generation. Standalone ec.balance is unaffected.
* fix(ec): re-sweep recovered nodes before generation so they aren't stranded
A node skipped as down by the pre-encode sweep is excluded from the rebalance,
but it can recover and become the generation host — mounting all shards locally,
then being excluded from distribution. Union-only verification accepts all
shards on one node and deletes the originals: a single point of failure. Re-sweep
the skipped nodes just before generation; one whose teardown now succeeds leaves
the skipped set and rebalances normally, while a node still down stays skipped.
* fix(ec): abort the encode if a selected source is still skipped after re-sweep
The re-sweep un-skips a recovered node, but the source was selected before it and
a node can stay down through the re-sweep then recover just in time to be the
generation host — mounting all shards locally while still excluded from the
rebalance, which union-only verification accepts before deleting the originals.
Abort the encode when a selected source remains skipped after the re-sweep.
* fix(ec): batch delete returns retriable 503 when a volume became EC mid-batch
If a volume is not EC at the batch-delete classification but is encoded to EC and
its .dat deleted before the regular-volume mutation, the mutation returns an exact
"not found" that the filer chunk-GC treats as completed, dropping the delete.
Recheck EC presence under the mutation lock and return a retriable 503 with the
"try again" token so the filer requeues it onto the EC path.
* fix(ec): recheck EC state before the regular batch-delete mutation
ec.encode mounts EC shards (copied from the .dat) before deleting the originals,
so a volume can be EC while its .dat still exists. The batch delete only rechecked
EC after a NotFound, so a successful regular-volume delete in that window wrote a
tombstone to the soon-removed .dat — the delete was lost and the needle resurrected
from the pre-tombstone shards. Recheck has_ec_volume under the write lock before
delete_volume_needle and return a retriable 503 so the filer requeues onto the EC path.
* fix(volume): make the metrics push test independent of test order
test_push_metrics_once asserted the pushed body contains the request-counter
family without ever touching the counter — a CounterVec with no children emits
nothing, so the assertion only held when another test had already created a
labelset in the shared registry. Create one in the test itself.
On volumes failing scrubs, the detail output can get very verbose, which makes
reading results difficult. Most users won't care about this information to
begin with - just whether or not volumes pass scrub tests.
This MR gates the display of scrub result details behind a `--details` flag.
* ci(s3tables): drop docker pre-pull from Lakekeeper job
The lakekeeper repro is pure Go against the local weed binary; the job
kept failing on Docker Hub timeouts pulling python:3 and localstack
images the test never runs. Also drop the stale python-in-docker
comments left from the old harness.
* ci(s3tables): serve python:3 from GHA cache in the STS job
Retried pulls still die when both mirror.gcr.io and registry-1.docker.io
are unreachable from the runner. Cache the saved image tarball under a
weekly key: an exact hit skips the registry entirely, a miss pulls fresh
and refreshes the cache, and a stale tarball from a previous week is the
fallback when Docker Hub is down.
* ci(spark): pre-pull the spark tag the test actually runs
The workflow warmed apache/spark:3.5.8 with retries while the
testcontainers setup runs apache/spark:3.5.1, so the real image was
pulled at test time with no retry at all.
* Filter metadata events before unmarshaling them per subscriber
Every subscriber unmarshaled every log entry into a full event just to
run the path filter, and entries carry complete chunk lists, so a fleet
of path-filtered subscribers spends almost all replay CPU materializing
events it then discards. A shallow wire scan now extracts just the
directory, entry names and rename destination into a skeleton event,
feeds the same matcher, and skips the decode for entries the subscriber
cannot match. Any scan surprise (malformed bytes, merged duplicate
message fields) falls back to the full decode, and the unsynced-events
heartbeat keeps firing for skipped entries.
* Raise the legacy replay cap
The cap was sized when every replay pinned a private chunk reader per
source filer. Replays now share decoded chunks, so sixteen needlessly
serializes subscriber catch-up; the expensive part stays bounded by the
cache's load gate.
* Weight concurrent log-chunk loads by size
The flat eight-load gate let eight tiny chunks through as reluctantly as
eight full ones. Charge each load's chunk size against a 128MB in-flight
budget instead: small chunks decode wide open while full-size ones still
serialize enough to cap the transient peak. Oversized weights clamp to
the budget so they can always acquire.
* Propagate heartbeat send failures and reset the skip counter
A failed heartbeat send means the stream is gone, so end the replay
instead of scanning on. A delivered event also resets the skip counter,
keeping the heartbeat cadence relative to the last thing the client
actually received.
* Share the unsynced-events counter across the prefilter and delivery
Two independent counters could starve the heartbeat: alternating drops
reset each side before either reached its threshold. One shared counter
increments on every dropped entry, prefiltered or not, and only an
actual delivery resets it, restoring the original cadence exactly.
* Tighten comments
* Benchmark the subscription match paths
For a thousand-chunk event that the subscriber filters out, the shallow
scan matches in 10us and 9 allocations against 175us and 4031
allocations for the full decode.
The admin UI served embedded static files uncompressed and without
cache headers: embed.FS has zero mod times, so no Last-Modified, no
ETag, no 304s -- every page load re-downloaded ~700KB of css/js in
full, which gets painful over slow or tunneled links.
Gzip the static tree at generation time (go generate ./weed/admin)
and embed only the compressed mirror, shrinking the binary ~1.5MB.
The handler hands the pre-compressed bytes to gzip-capable clients,
decompresses for the rest, and sets Cache-Control, per-variant
content-hash ETags and Vary so repeat loads revalidate with a 304.
bootstrap.min.css goes 232KB -> 30KB on the wire.
A drift test keeps static_gz/ in sync with static/.
The eachLogDataFn error path printed the full LogEntry proto. For an
entry carrying a large chunk manifest that is hundreds of KB of escaped
bytes in a single log line, burying the actual error -- often just a
subscriber disconnect -- at the very end. Log the key, timestamp,
offset and data size instead.
* elastic: fix listing against a missing or empty directory index
The refresh 404 leaked into the named return, so the first listing of a
directory whose index does not exist yet returned an error instead of an
empty result. Sorting also fails on an index with no documents
("No mapping found for [_id] in order to sort on"); unmapped_type
keeps the resumed-listing path working there.
* add elastic8 filer store for Elasticsearch 8
Elasticsearch 8 disables _id fielddata by default, so the elastic7
store's directory listings fail with "Fielddata access on the _id
field is disallowed". elastic8 uses the same client and configuration
options, but also indexes the document id as an Id field and sorts
listings on Id.keyword.
Problem: Signature V4 SignedHeaders parsing accepted empty header name segments such as host; or ;host. Malformed Authorization headers could continue into signature verification instead of failing during header parsing.
Root cause: parseSignedHeader only checked that the SignedHeaders value was non-empty, then split it on semicolons without validating each element.
Fix: reject empty or whitespace-only signed header elements with ErrMissingFields before returning the parsed header list.
Reproduction: go test ./weed/s3api -run TestParseSignedHeaderRejectsEmptyHeaderNames -count=1 failed before the fix because SignedHeaders=host; returned ErrNone.
Validation: gofmt -w weed/s3api/auth_signature_v4.go weed/s3api/auth_signature_v4_test.go; git diff --check; go test ./weed/s3api -run TestParseSignedHeaderRejectsEmptyHeaderNames -count=1; go test ./weed/s3api -count=1
Co-authored-by: Codex <noreply@openai.com>
* Bound the metadata-log flush queue
A stalled flush, e.g. slow volume servers under a reconnect storm, let up
to 256 queued 8MB buffer copies pin two gigabytes per log buffer while
producers kept filling the queue. Cap the queue at 16 so a sustained
stall backpressures writers instead of growing the heap. The flush
goroutine never feeds back into the buffer (system-log paths skip event
notification), so blocked producers cannot deadlock the consumer.
* Don't drop a force-flushed buffer on a full queue
ForceFlush enqueued with a two-second timeout, but by then the live
buffer was already sealed and reset, so a timed-out send silently lost
the copy. Block until the flush is queued; the wait for completion stays
bounded since the data is durable once the flush loop drains it.
* Never close the flush channel
ShutdownLogBuffer closed flushChan while producers could still be
blocked sending into it, which panics. Terminate loopFlush with a nil
sentinel instead, so the channel is never closed, and give every
producer-side send a shutdown escape so none parks forever once the
flush loop exits. Everything queued before the sentinel still drains,
preserving IsAllFlushed semantics.
* Copy the shutdown flush under the buffer lock
Every other copyToFlush call site holds the lock; the shutdown path read
the live buffer unlocked while producers could still be appending.
* Share metadata-log replays per chunk instead of per file
Log file chunks are immutable: each metadata-log flush uploads one whole
buffer of complete records as a new chunk, and appends only add chunks.
So cache decoded entries per chunk, with no age gate and no fingerprint
revalidation. The per-file cache excluded files younger than two flush
intervals, which is exactly the hot tail that every tailing or
reconnecting subscriber replays — each through a private chunk reader
holding an 8MB buffer and decoding the whole file from byte zero.
A chunk's flush time also upper-bounds every record timestamp inside it,
so a tail replay now skips cold chunks without reading them at all.
If a chunk does not decode standalone (records spanning chunk
boundaries, or a corrupt size prefix), fall back to streaming the whole
file as one byte stream, resuming after the last yielded entry.
* Evict idle metadata-log cache entries
The replay cache only evicted on insert, so once filled it held its full
budget forever. Stamp entries on use and sweep the LRU tail every minute,
dropping anything untouched for five minutes; the cache now holds memory
only while subscribers actually replay.
* Reject implausible records when decoding log chunks
proto.Unmarshal is permissive: empty payloads and unknown-field garbage
parse without error, so a chunk starting mid-record could decode by
coincidence and get cached instead of falling back to the byte stream.
Enforce what the writer guarantees - records are never empty and carry
strictly increasing positive timestamps within one flushed buffer.
* Gate the singleflight test on an open flight
The sleep alone only probabilistically created concurrent misses; a
started channel now proves the loader holds the flight before callers
are released.
Problem: SigV4 chunked upload checksum trailer parsing rejected mixed-case checksum header names even though HTTP header field names are case-insensitive.
Root cause: extractChecksumAlgorithm compared the x-amz-trailer value and trailer header key against exact lowercase strings.
Fix: Trim and lowercase checksum trailer header names before matching supported checksum algorithms.
Reproduction: go test ./weed/s3api -run TestExtractChecksumAlgorithmIsCaseInsensitive -count=1 with X-Amz-Checksum-Crc32; before the fix it returned unsupported checksum algorithm.
Validation: gofmt -w weed/s3api/chunked_reader_v4.go weed/s3api/chunked_reader_v4_test.go; git diff --check; go test ./weed/s3api -run TestExtractChecksumAlgorithmIsCaseInsensitive -count=1; go test ./weed/s3api -count=1
Co-authored-by: Codex <noreply@openai.com>
* ci(s3tables): route Docker Hub pulls through mirror, drop unused buildx
The integration jobs set up docker/setup-buildx-action only to docker
pull/run images; the buildx bootstrap pulls moby/buildkit from
registry-1.docker.io, which times out and fails the whole job before any
test runs. These jobs never docker build with buildx, so the setup is
pure overhead and an extra registry dependency.
Replace it with a daemon registry-mirror pointing at mirror.gcr.io (a
pull-through cache for Docker Hub) and retry the pre-pulls a few times.
That removes the buildkit pull entirely and routes the rest through the
cache, with graceful fallback to Docker Hub on a miss.
* ci: route Docker Hub through mirror in remaining docker test workflows
Same registry-1.docker.io timeout fix for the other integration jobs.
s3-spark only docker pulls/runs an image, so drop the vestigial buildx
setup and pull through the mirror with retries, matching s3-tables.
kafka-quicktest, s3-proxy-signature, e2e and postgres build/compose and
genuinely need buildx (e2e/postgres export a local layer cache, which the
default driver can't), so keep it and just configure the mirror first —
that way even the moby/buildkit bootstrap pull is served from the cache.
Left samba/pjdfstest alone: they build-push to a local registry and pull
from localhost, so buildx is required and there's no Docker Hub runtime
pull to mirror.
perf(filer): share decoded log entries across metadata replays
Concurrent SubscribeMetadata replays of the same persisted log history each
opened a chunk reader per source filer and re-decoded the same files, so a
reconnect storm multiplied into many GB of buffers. Cache the decoded entries
of completed log files in a bounded LRU, coalescing concurrent loads with
single-flight and bounding concurrent decodes. Each hit is validated against
the file's current chunk set, so a file that received a late append is
reloaded rather than served stale; reads that stop on an unreachable chunk are
delivered but not cached so a transient outage re-probes on the next replay.
* fix(filer.sync): replicate a rename as create-then-delete, not an in-place update
A rename arrives as a single metadata event carrying both the old and new
entry. The filer sink was routed to UpdateEntry, which looks up the old
path but issues the update against the new parent without changing the
name — and the filer UpdateEntry RPC cannot move an entry. So the rename
was dropped: the old path lingered and the new path never appeared
(same-dir renames rewrote the old name in place).
Route a real move (the sink path changed) through CreateEntry(new) then
DeleteEntry(old) in both the replicator and the filer.sync/backup driver,
the way the other sinks already handle it; reach UpdateEntry only for true
in-place updates. Create before delete so a crash between the two leaves
the entry visible rather than lost.
* fix(filer.sync): derive the rename delete key like the create key, guard the watched root
The rename delete leg rebuilt the old key with a raw util.Join, bypassing the
sink-side key normalization the create leg gets from buildKey — so a rename
could create the new entry and then fail to delete the old one under a
transformed key. Build the old key through buildKey too, and skip the delete
when the moved entry is the watched root itself (where the old key would
resolve to the target root and recursively delete the whole sink tree).
* test(filer.sync): cover the in-place update delete-then-create fallback order
The recording sinks always reported foundExisting, so the fallback that an
in-place update takes when the entry is missing on the sink was never run.
Make it configurable and assert the fallback deletes before it recreates the
same key, in both the replicator and the filer.sync drivers.
* feat(filer.sync): move filer-sink renames natively via AtomicRenameEntry
create-then-delete is unsafe for the filer sink: CreateEntry returns nil
without creating on a transient chunk-copy error, so the paired delete could
remove the only valid destination copy; a directory rename also deleted the
old subtree before descendants were recreated, and left old chunks behind.
Add an optional EntryMover sink capability and implement it on the filer sink
via AtomicRenameEntry — one atomic, metadata-only move that relocates a whole
subtree in a single transaction. Renames prefer it; sinks without a native
move keep create-then-delete. When the old path is already gone (a descendant
the parent rename moved, or one never replicated) MoveEntry creates the new
path instead, re-checking existence with a lookup so a rolled-back move that
left the old entry intact is retried rather than mistaken for gone.
* docs(filer.sync): note entryMissing's gRPC not-found string fallback is deliberate
* fix(http): accept no content delete responses
Problem: util/http.Delete reports an error for a successful HTTP 204 No Content response.
Root cause: Delete only treats 200 OK, 202 Accepted, and 404 Not Found as non-error responses, omitting the standard 204 status commonly returned by DELETE endpoints.
Fix: Include http.StatusNoContent in the Delete success status set.
Reproduction: go test ./weed/util/http -run TestDeleteTreatsNoContentAsSuccess -count=1 fails before the fix with an empty error for a 204 response.
Validation: go test ./weed/util/http -run TestDeleteTreatsNoContentAsSuccess -count=1; go test ./weed/util/http -count=1; git diff --check; git diff --cached --check
Co-authored-by: Codex <noreply@openai.com>
* Update weed/util/http/http_global_client_util_test.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix(filer): propagate multi-filer metadata log read errors
A genuine (non not-found) read error in one filer's log stream was logged
and skipped, then the merged cursor advanced past the gap, silently
dropping that file's events. Abort the whole replay so the subscriber
re-reads from the unchanged position; chunk-not-found still skips.
* perf(mount): read persisted metadata log chunks directly from volume servers
Set LogFileReaderFn so the filer returns log file references and the mount
reads the chunk data itself, instead of the filer reading, decoding, and
streaming every persisted entry. Keeps a reconnect storm of many mounts
from concentrating hundreds of concurrent log replays in filer memory.
* perf(filer): pre-size chunk stream reader buffer to view size
The chunk size is known up front, so grow the buffer once instead of
letting bytes.Buffer double as the streamed pieces arrive (which
transiently overshoots to ~2x per reader).
* fix(filer): bound concurrent persisted-log replays
Each server-side replay holds an open chunk reader per source filer plus a
readahead buffer, so a reconnect storm of clients that predate the
metadata-chunks offload multiplies into many GB. Gate replays with a
semaphore; abort the acquire when the subscriber's stream is gone so
cancelled clients do not pile up parked goroutines.
* feat(filer): record object size distribution histogram
Add SeaweedFS_filer_object_size_bytes, a histogram sampled when an
object is first created in the filer namespace, covering every write
protocol (S3, WebDAV, FUSE mount, direct HTTP). Buckets follow the
1KB/100KB/1MB/100MB/1GB ranges operators use to size collections.
Directories, overwrites, and metadata-only updates are not sampled, so
the bucket counts track the size distribution of distinct objects.
* feat(metrics): add filer object size distribution dashboard panels
Add a write-rate-by-size-range graph and a size-distribution bar gauge,
driven by SeaweedFS_filer_object_size_bytes, to the standalone and Helm
Grafana dashboards. Per-range subtractions are clamped at zero so
transient negative rate() samples do not render below the axis.
* fix(filer.sync): scope filesystem key sanitization to the local sink
destKey ran every sink key through escapeKey, whose Windows build strips
colons. Colons are illegal in NTFS filenames so the local sink needs that,
but s3/filer/azure/gcs/b2 accept them as ordinary key bytes — stripping
them silently diverged the destination key (a source a:b replicated as ab).
Move the sanitization into the local sink behind a Windows build tag,
applied at every entry point so the previously-unescaped in-place-update
paths stay consistent. Non-local sinks now keep the raw key; non-Windows
builds are unchanged; a leading drive-letter colon is preserved.
* test(filer.sync): cover incremental destKey and localsink update/delete sanitization
Lock the colon-preserving behavior for the incremental destKey branch, and
extend the Windows local-sink test to assert UpdateEntry and DeleteEntry also
sanitize the key, not just CreateEntry.
* fix(filer.backup): skip replay events whose chunk no longer exists on the source
"Source" is the filer we replicate FROM (e.g. green in a green->blue backup).
Replaying the metadata log from a checkpoint can hit an event whose chunk was
since overwritten/deleted and garbage-collected on the source volume. Fetching
it returns 0 bytes (a permanent size mismatch), which the sink propagated to the
subscription — so the same offset retried forever and replication stalled.
Skip the event only when proven stale; otherwise keep refusing so genuine loss
of a live file still halts loudly:
- onCorruptChunk centralizes the three errChunkSizeMismatch sites.
- getEntryMtimeNs compares mtime at nanosecond precision so same-second rewrites
(git's config.lock dance) are ordered correctly.
- sourceSupersedes re-reads the entry's current state on the source: gone
(ErrNotFound) or a strictly-newer mtime than the replayed version -> skip;
any other lookup error keeps the entry.
Skipping is lossless: events are full-entry snapshots, so a later event
re-carries the current chunks and a delete event reconciles a removed file.
* test(filer.backup): cover the superseded-chunk skip decision
- TestSourceSupersedes: not-found (sentinel / wrapped / gRPC string) and nil
entry -> skip; network error -> keep; source newer -> skip; same/older -> keep.
- TestGetEntryMtimeNs: nanosecond precision, same-second ordering, nil safety.
- TestOnCorruptChunkRefusesWhenSupersessionUnconfirmed: never skip silently when
supersession cannot be confirmed.
* fix(filer.backup): don't infer supersession for incremental sinks
In incremental mode the sink key carries a date prefix
(sinkDir/YYYY-MM-DD/relPath) that cannot be reversed to a real source path, so a
source lookup would always be ErrNotFound and wrongly classify a live entry as
deleted — skipping it. Make targetPathToSourcePath report "unmappable" in
incremental mode; hasSourceNewerVersion already declines to skip when the source
path cannot be mapped.
Found in code review. Non-incremental sinks (filer.backup green->blue) are
unaffected.
* refactor(filer.backup): name the mtime param sourceMtimeNs; note ns overflow bound
- Rename the threaded sourceMtime parameter to sourceMtimeNs across the internal
replicate/fetch helpers so the unit is explicit (it only feeds
hasSourceNewerVersion, which compares in nanoseconds).
- Document that getEntryMtimeNs's int64 ns arithmetic is safe until ~year 2262.
No behavior change.
* fix(filer.backup): order same-second versions in the CreateEntry skip and update gates
The CreateEntry already-replicated short-circuit and chooseUpdateAction
still compared second-grained mtime, so a newer version written within
the same second could be skipped as already-replicated or overwritten by
an older same-second replay. Route both through getEntryMtimeNs, matching
the precision the chunk-replication path already uses.
* test(filer.backup): cover same-second update-action ordering
* docs(filer.backup): trim verbose comments to terse why
* fix(filer.backup): check supersession against the rename's new path
For a rename the filer sink updates in place (the delete+create branch is
skipped for sink name "filer"), so the corrupt-chunk supersession check
queried the pre-rename key. Its source-side ErrNotFound was read as
"superseded", silently advancing the checkpoint without applying the rename.
Map the incoming entry's new path (newParentPath/newEntry.Name) for both
update branches.
* fix(filer.backup): detect a deleted source even when the replayed mtime is epoch
hasSourceNewerVersion returned early when sourceMtimeNs <= 0, skipping the
source lookup, so a deleted entry with mtime 0 (a valid epoch timestamp) never
got the gone verdict and wedged on permanent retries. Always look up; gate only
the newer-mtime comparison on a valid replayed mtime.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Problem: Re-storing object-lock default retention with Days left a previous Years extended attribute in place, so later loads could see both Days and stale Years.
Root cause: StoreObjectLockConfigurationInExtended only wrote period fields that were set on the new configuration and did not delete old Days or Years keys before writing the replacement rule.
Fix: Clear stored default-retention Days and Years keys before writing the current default retention period fields.
Reproduction: go test ./weed/s3api -run TestStoreObjectLockConfigurationClearsStaleYears -count=1 failed before the fix because the stale years key remained.
Validation: go test ./weed/s3api -run TestStoreObjectLockConfigurationClearsStaleYears -count=1; go test ./weed/s3api -count=1; git diff --check; git diff --cached --check
Co-authored-by: Codex <noreply@openai.com>
* fix(mongodb): merge URI auth fields with username/password override
SetAuth replaced the whole Credential parsed from the URI, dropping
AuthSource and AuthMechanism. Start from the URI-parsed Auth and only
override the username and password so credentials scoped to a specific
auth database keep working.
* fix(mongodb): set PasswordSet for explicit credentials
Required by GSSAPI auth when a password is supplied; ignored for other
mechanisms.
processExplicitDirectory appended a directory-key object as a version
without checking it against the prefix. A versioned listing descends
through ancestor markers to reach a deeper prefix, so every ancestor
(Veeam/, Veeam/Backup/, ...) leaked into Versions even though none of
them match the prefix - which makes Veeam's immutable repository scan
abort on an unexpected key. Guard on the prefix so only keys at or under
it surface, matching ListObjectsV2 and AWS.
* fix(shell): move files into existing destination directories
Problem: fs.mv /src/file /dst/dir treats an existing destination directory as a destination file path, so it renames the source to /dst/dir instead of moving it into /dst/dir/file.
Root cause: commandFsMv builds the destination LookupDirectoryEntryRequest with Directory and Name swapped, so the destination directory lookup misses.
Fix: Populate LookupDirectoryEntryRequest with Directory=destinationDir and Name=destinationName before deciding whether the destination is a directory.
Reproduction: env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/shell -run TestFsMvMovesIntoExistingDestinationDirectory -count=1
Validation: gofmt -w weed/shell/command_fs_mv.go weed/shell/command_fs_mv_test.go; git diff --check; git diff --cached --check; env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/shell -run TestFsMvMovesIntoExistingDestinationDirectory -count=1; env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/shell -count=1
* Update weed/shell/command_fs_mv_test.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix(seaweed-volume): stop EC shard deletion from phantom .dat on restart
On startup load_existing_volumes() scans .vif/.idx entries (not just
.dat). For distributed EC, a volume's .vif can be mirrored onto a disk
whose .ecx lives on a sibling disk, so the per-disk ecx check is false
and the loader falls through to Volume::new, which always creates the
.dat if missing -> a phantom 8-byte superblock stub. The store-level
prune_incomplete_ec_with_sibling_dat then treats that stub as the
authoritative source and deletes the real EC shards on sibling disks. Go
guards the same case (disk_location.go: 'Without this guard NewVolume
below would create a phantom empty .dat') but only same-disk.
Fix A (root cause): in load_existing_volumes, don't create a .dat during
load. Skip the entry when there is no local .dat AND the .vif does not
reference remote files -- remote-tiered volumes have no local .dat but
must still load via the remote path. Uses the robust check_dat_file_exists
helper so a transient stat error doesn't skip a real volume. New volumes
go through create_volume(). Covers the cross-disk .vif/.ecx split Go's
same-disk hasEcxFile() misses.
Fix B (defense in depth, Go + Rust): when the EC .vif records no source
size (dat_file_size==0), require the sibling .dat to be strictly larger
than a bare superblock, so an empty 8-byte stub can't pass the
credibility gate. Previously it fell back to SUPER_BLOCK_SIZE, which an
8-byte stub exactly meets.
Adds regression tests reproducing the cross-disk lone-.vif phantom and
the 8-byte stub gate; updates an existing prune test to use a real
collection so its .ecx lookup matches the loaders.
* fix(storage): don't create phantom .dat from lone .vif on Go volume load
Mirror Fix A on the Go side. loadExistingVolume scans .vif/.idx entries,
and for distributed EC a .vif can be mirrored onto a disk whose .ecx is
on a sibling disk. The same-disk hasEcxFile() guard does not fire there,
so the loader falls through to NewVolume(createDatIfMissing=true) and
writes an 8-byte phantom .dat, which the sibling-.dat prune then uses to
delete the real EC shards on sibling disks. Skip the entry when there is
no local .dat AND the .vif has no remote file (via MaybeLoadVolumeInfo);
remote-tiered volumes have no local .dat but must still load.
Adds TestLoneVifDoesNotCreatePhantomDat (fails without the guard) and
TestRemoteTier_DiskScanLoadsRemoteOnlyVolume (fails if the guard skips a
remote-only volume).
Problem: RandomUint64 generated eight random bytes but returned int32, truncating the value before mount file and directory handles converted it to uint64. This reduced handle entropy to 32 bits and produced sign-extended handle values.\n\nRoot cause: the helper cast BytesToUint64 to int32 and exposed int32 as its return type.\n\nFix: make RandomUint64 return uint64 and return the full BytesToUint64 result.\n\nReproduction: go test ./weed/util -run TestRandomUint64ReturnsUint64 -count=1 failed before the fix because RandomUint64() had kind int32.\n\nValidation: gofmt -w weed/util/bytes.go weed/util/bytes_test.go; git diff --check; go test ./weed/util -run TestRandomUint64ReturnsUint64 -count=1; go test ./weed/util -count=1; go test ./weed/mount -count=1; git diff --cached --check
Adds an "Export All (JSON)" button on the Cluster Volumes page that pulls
the whole cluster's volume list from the master in one call, a superset of
volume.list. Beyond the table columns it carries garbage and fullness
ratios, modified time, compact revision, remote tiering keys, per-disk
capacity counts, EC shard sizes with file/delete counts, and a cluster-wide
duplicate-volume-id scan. Honors the active collection filter. The existing
per-page CSV export stays as "Export Page".
Mini auto-sizes a few large volume slots, but the master pre-grows 7
volumes per new collection. Under a filer group each S3 bucket is its
own collection, so the first buckets claimed every slot and later
writes failed to assign a volume. Cap mini's volume_growth copy counts
to 1.
* fix(filer.sync): guard batched events against nil EventNotification
The server folds a backlog into one response: the first event in the
top-level fields, the rest in resp.Events, and the pipelined sender can
drain an idle heartbeat (nil EventNotification) into that tail. Only the
envelope got the freshness-signal guard, so a batched heartbeat reached
AddSyncJob and nil-derefed in IsEmpty while replaying a backlog buffered
during a peer outage.
Route every event, envelope and batched, through one handler that sends
freshness signals (nil heartbeat, empty marker) to OnIdleHeartbeat.
* fix(filer): guard MetaAggregator batched events against nil EventNotification
The peer subscription's envelope is nil-guarded but its batched tail was
not. The aggregator doesn't enable idle heartbeats today, so the server
can't fold a nil EventNotification into the batch yet, but make the two
loops consistent so it can't nil-deref if that changes.
fix: don't mangle filer paths with the OS separator on Windows
filepath.Dir/Join use the platform separator, so on Windows they rewrite
a forward-slash filer path like /buckets/x into \buckets\x. The mangled
value then goes into a filer RPC and operates on the wrong key, so the
op silently targets nothing.
The admin file browser hit this in New Folder (the entry landed under
\buckets\my-bucket and never showed up under /buckets/my-bucket), and
the same way in delete, view and properties. MQ topic retention and
consumer-offset listing, and the SFTP home dir plus create-permission
parent lookup, had the same bug.
Switch all of these to the path package, which always uses "/".
* fix(remote): reject short reads when caching remote objects
A short read from the remote (stale listing size, truncated or flaky
response) was silently zero-padded: the S3 and Azure clients pre-size
the buffer and discard the downloaded byte count, and the chunk is
recorded with the requested size. The cached file then matched the
expected size but its tail was NULL, and the entry was marked cached
so it never re-fetched.
Check the byte count against the requested size in both clients, and
add a backend-agnostic guard in FetchAndWriteNeedle. The cache now
fails loudly and the entry stays remote-only for a later retry.
* fix(remote): match S3 default modes when syncing remote metadata
Remote object listings carry no POSIX mode, so synced entries were
created with a hardcoded 0644. Against a SeaweedFS remote, whose S3
layer writes objects as 0660 and auto-creates directories as 0771
(0660|0111), the mounted copy ended up 0644/0755 and the permissions
visibly diverged from the source.
Default to the S3 modes instead (files 0660, directories 0771). The
filer derives parent-dir modes from the child as fileMode|0111, so
fixing the file default also brings the directories into line.
Directory mtimes still reflect sync time: S3 listings don't enumerate
directories, so the remote's directory timestamps aren't available.
TopicExists and getTopicConfFromCache negative-cached a topic for the full
30s TTL whenever a filer lookup failed for any reason, including timeouts.
A topic created earlier then looked gone until the TTL expired, and the
metadata auto-create path couldn't heal it (CreateTopic rejects an
already-persisted conf), so producers saw UNKNOWN_TOPIC_OR_PARTITION.
Only negative-cache on a definitive ErrNotFound; let transient errors fall
through and retry against the filer.
* fix(s3api): accept HTTP-date conditionals
Problem: Object conditional headers rejected valid HTTP-date values in RFC850 or ANSIC format for If-Modified-Since and If-Unmodified-Since.
Root cause: parseConditionalHeaders used time.Parse(time.RFC1123), accepting only one HTTP-date representation instead of the standard formats accepted by net/http.ParseTime.
Fix: Parse conditional date headers with http.ParseTime so RFC1123, RFC850, and ANSIC HTTP-date forms are accepted.
Reproduction: go test ./weed/s3api -run TestParseConditionalHeadersAcceptsHTTPDateFormats -count=1 failed before the fix with ErrInvalidRequest for RFC850 and ANSIC date values.
Validation: env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/s3api -run TestParseConditionalHeadersAcceptsHTTPDateFormats -count=1; env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/s3api -count=1; git diff --check; git diff --cached --check
* fix(s3api): accept HTTP-date copy-source conditionals
Mirror the put-path http.ParseTime switch onto the copy-source If-Modified-Since / If-Unmodified-Since headers, which still rejected valid RFC850 and ANSIC dates.
* fix(s3api): keep RFC1123 UTC-zone dates working alongside http.ParseTime
http.ParseTime rejects the "UTC" zone that Go clients emit via t.UTC().Format(time.RFC1123), which the old RFC1123 parser accepted. Add a parseHTTPDate helper that tries http.ParseTime first and falls back to RFC1123, so the put and copy-source conditional date headers accept the union of HTTP-date formats plus the UTC zone.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat(filer.backup): add -resetCheckpoint to force a fresh sync
filer.backup resumes from a per-sink offset persisted in the source filer's KV.
There was no first-class way to discard that checkpoint and re-run from the
beginning short of guessing a large -timeAgo, which also skips -initialSnapshot.
Add -resetCheckpoint: before reading the offset, write 0 for this sink so
getOffset returns 0, isFreshSync stays true, and -initialSnapshot re-runs a full
walk. Effective only when -timeAgo is 0.
The flag is cleared after the first successful reset: runFilerBackup retries
doFilerBackup forever on error, so leaving it set would re-zero the checkpoint
on every retry and never make forward progress after a transient failure. Later
retries resume from the persisted checkpoint instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(filer.backup): keep fresh-sync intent when offset read fails after reset
After -resetCheckpoint writes offset 0, a transient getOffset read-back error
flipped isFreshSync to false, which skipped the -initialSnapshot walk the reset
explicitly requested. Track that the reset happened this iteration and, on a
getOffset error, preserve isFreshSync=true in that case (the non-reset path
keeps treating a read error as "not fresh" to avoid re-walking on transients).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(filer.backup): skip offset read-back on reset instead of tracking a flag
Replace the didReset bool by branching: on -resetCheckpoint, clear the offset and
start fresh without reading it back (we just wrote 0, so the state is known);
otherwise read the offset as before. This drops the redundant getOffset RPC after
a reset and removes the read-back error case entirely, so no separate flag is
needed to preserve isFreshSync.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* filer.backup: -initialSnapshot re-seeds on every start; drop -resetCheckpoint
-initialSnapshot now walks the live tree whenever -timeAgo is 0, seeds the
destination, and overwrites the saved checkpoint, rather than running only on a
fresh sync. That re-seeds a reinitialized destination on its own, so the
separate -resetCheckpoint flag is gone.
The walk runs once per process: the in-memory flag is cleared after the
watermark is persisted, so the retry loop resumes from the persisted checkpoint
instead of re-walking on every transient error. A process restart re-walks, so
remove the flag once the backup is caught up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust release: publish .md5 checksums alongside weed-volume binaries
The versioned rust volume release built and uploaded the tarballs/zips but
no checksum sidecars (the Go releases get .md5 automatically via
go-release-action; this workflow uses softprops/action-gh-release directly).
Generate an .md5 next to each asset (md5sum on linux/windows-bash, md5 -r on
macOS) and include them in the release/artifact uploads, so downloaders
(e.g. seaweed-up, which verifies md5 before installing weed-volume) can
check integrity. Covers linux amd64+arm64, darwin amd64+arm64, windows amd64.
* rust release: build large-disk and normal into separate target dirs
Both cargo builds wrote to target/<triple>/release/weed-volume, so the
second (normal, --no-default-features) overwrote the first, and the Package
step then copied that same binary into BOTH tarballs — the large-disk asset
actually shipped the normal binary. Build each variant into its own
--target-dir (target/large-disk and target/normal, both under target/ so the
existing cache still covers them) and copy each tarball's binary from its
own dir.
* admin/maintenance: reload in-flight tasks on startup instead of discarding
LoadTasksFromPersistence deleted all persisted task files on startup and
relied on the scanner to re-detect, so saved task state was never consumed
— the persistence was effectively write-only. Reload non-terminal tasks
(pending/assigned/in_progress) into the queue, resetting in-flight ones to
pending since their worker is gone after a restart (maintenance tasks are
idempotent). Terminal task files are dropped; the scanner still backfills
anything not persisted.
* address review: nil-guard reloaded tasks and SyncTask to ActiveTopology
- skip nil entries from LoadAllTaskStates (corrupted state)
- re-sync restored tasks with MaintenanceIntegration so ActiveTopology
(in-memory, empty on startup) knows about them; otherwise GetNextTask's
AssignTask rejects them as unknown and they never get assigned
* vacuum: compact a read-only volume when an explicit volumeId is given
The on-demand path no longer skips read-only volumes, so an operator can
reclaim a benignly read-only (full/oversized) volume without marking it
writable first. The background scan and all-volumes sweep still skip
read-only, where the flag usually signals an unhealthy disk.
* vacuum: copy locationList under lock for on-demand vacuum
The volumeId>0 path passed the live vid2location entry into the async
vacuum, where heartbeat-driven Register/UnRegister can mutate the slice
concurrently. Snapshot it under accessLock, matching the sweep path.
* ec.balance: verify shard(s) landed on the destination before deleting source
The EC balance task copied/mounted a shard to the destination and then
immediately unmounted+deleted it from the source, reporting success as soon
as the RPCs returned. A copy/mount can return OK while the shard isn't
actually registered/loadable on the destination, so deleting the source
then loses the shard (and the scanner re-issues the same move every cycle).
Add a verification step (VolumeEcShardsInfo via VerifyShardsAcrossServers,
the same check the EC encode task uses before deleting originals): if the
destination doesn't report every moved shard, fail the task and keep the
source so the move is retried instead of losing data.
* address review: use comma-ok when reading destination shard inventory
Problem: Default object-lock retention accepted an explicitly provided Years value of zero, even though a default retention period must be positive when present.
Root cause: validateDefaultRetention rejected zero Days but only rejected negative Years, leaving YearsSet with Years=0 as a successful validation path.
Fix: Treat an explicitly provided zero Years value as ErrInvalidRetentionPeriod, matching the existing Days validation.
Reproduction: go test ./weed/s3api -run TestValidateDefaultRetention -count=1 failed before the fix because the Zero years case returned nil.
Validation: go test ./weed/s3api -run TestValidateDefaultRetention -count=1; go test ./weed/s3api -count=1; git diff --check; git diff --cached --check
* ec placement: steer shards to less-loaded machines, not the lowest id
EC encode places every volume against one shared topology snapshot (it reserves the
shards it assigns so later volumes see reduced capacity), but node selection ranked
only by this volume's shard count and broke ties by sorted id. So the lowest-id
machine won the first shard of every volume and accumulated far more total shards
than the rest -- on a 6-machine cluster the first machines drifted to ~1.5x.
Rank eligible nodes by the machine's shards of this volume, then the machine's free
capacity, then the node's shards of this volume, then the node's free capacity. Free
capacity reflects the load already placed, so ties steer toward the least-loaded
machine instead of the lowest id, keeping total EC shards even across machines.
* test: ec.balance converges to even per-machine load from a skew
Starts machine 10.0.0.1 at 4 shards/volume and the rest at 2, then runs repeated worker-style capped passes; asserts convergence to an even per-machine total (reaches exactly even in ~13 rounds).
* reduce comments on the placement fix
Trim narration to the non-obvious why.
* test: assert convergence and count zero-shard machines
Seed the per-machine map with every host so a fully drained machine still registers, and fail explicitly if balance doesn't converge before the round cap.
admin: default -dataDir to "." so maintenance task state persists
Previously -dataDir defaulted to empty, so the admin ran maintenance in
memory only: task state was never saved and maintenance tasks (notably EC
balance/rebuild) were re-issued every scan cycle without converging,
churning EC shards (moves landed shards without their .ecx index, leaving
EC volumes unloadable/missing shards).
Default -dataDir to "." (the process working directory, which under the
standard systemd unit is the admin's data dir) so state persists out of
the box.
* admin/topology: carry the volume server address on DiskInfo
The planning DiskInfo exposed only the node id, which can be an opaque label rather than ip:port. Record the address too so callers can resolve the physical machine a disk sits on.
* ec.balance: spread a volume's shards across machines, not just nodes
Volume servers sharing a host are one fault domain, but the within-rack spread treated them as independent nodes, so one box could end up holding more shards of a volume than EC can afford to lose. Add a machine (host) tier between rack and node: the within-rack pass spreads each volume across machines, and the global load phase no longer re-concentrates a volume onto a machine it already sits on. Host defaults to the node id, so clusters with one server per host are unchanged.
* ec placement: prefer machines holding fewer of a volume's shards
EC allocation and repair picked the least-loaded node in a rack with no regard for which physical machine it sits on, so a volume's shards could pile onto several servers of one box. Rank candidate nodes by their machine's shard count first, then the node's own. The machine is derived from the volume server address carried on DiskInfo, falling back to the node id, matching how the balancer resolves it.
* volume.balance: don't move a replica onto a machine already holding one
isGoodMove only rejected a move onto the same data node, so two replicas could land on two volume servers of one box and a single machine failure would lose both. Reject a target whose host already holds another replica of the volume. Best-effort: balancing simply skips and tries the next target.
* volume allocation: spread same-rack replicas across machines
PickNodesByWeight filled the same-rack replica picks by weight alone, so replicas could co-locate on one box. Prefer candidates on not-yet-used hosts, falling back when too few distinct machines exist. Data-center and rack tiers have no host, so their ordering is unchanged.
* ec.balance: harden machine spread against re-concentration and capped machines
Two cases where the machine-aware spread could still leave a volume badly placed:
- The global load phase could move a shard of a volume onto a machine that
already held it, raising that machine's count and undoing the within-rack
spread (a 4/4/3/3 layout could become 3/5/3/3, past parity for 10+4). Limit
the load-only fallback to same-machine moves, which leave a machine's count
unchanged; cross-machine concentration is no longer allowed for load alone.
- The within-rack spread chose a destination machine by free slots alone, so if
that machine's only nodes were already at the SameRackCount cap it skipped the
move instead of trying another machine. Require a machine to have a node that
can actually take the shard before selecting it.
* reduce comments across the machine-affinity change
Trim narration down to the non-obvious why; one terse line where a block was overkill.
* ec.balance: gate machine spread on fault-tolerance feasibility
Spreading a volume evenly across machines only helps when there are enough that
each can stay within EC's parity tolerance (numMachines >= ceil(total/parity)).
With fewer -- or wildly unequal -- machines it can't make a machine loss
survivable anyway, and forcing it fights capacity: e.g. a cluster of 12 volume
servers on one host and 2 on another would have half of every volume crammed onto
the 2-server box. So spread across machines only when it's achievable; otherwise
fall back to per-node spread and let capacity/global balancing decide.
The global load phase applies the same test: it protects a volume's machine spread
(no cross-machine move that raises a machine's count past the source's) only where
that spread is achievable, so heterogeneous clusters still level by fullness.
* ec.balance worker: group servers by host when planning
The worker built its planner topology without recording each server's host, so
automated ec.balance treated ports on one machine as independent nodes and could
concentrate a volume's shards on one physical box. Set the host from the volume
server address, matching the shell path.
* volume.balance worker: don't move a replica onto a machine holding one
The worker compared only node ids, and the replica map dropped the server address,
so it could move replicas onto different ports of one machine. Carry the host on
ReplicaLocation (from the server address) and reject a target whose host already
holds another replica of the volume. Best-effort, matching the shell.
* ec.balance: judge machine-spread feasibility by the rack's shards
The within-rack and global feasibility checks compared the whole volume's shard
count against a rack's machine count, so a rack holding only part of a volume after
cross-rack spreading -- e.g. 7 of a 10+4 volume across 2 machines -- was wrongly
judged infeasible and fell back to node spread, which could pile 6 shards onto one
host, past parity. Gate on the rack's own shard count of the volume instead.
* ec.balance: spread a volume's shards across machines by combined count
EC recovers from any loss within parity regardless of shard type, so what bounds a
machine's exposure is its total shards of the volume, not data and parity
separately. Spreading the two independently let each type's remainder land on the
same machine -- ceil(d/M)+ceil(p/M) can exceed ceil(total/M), e.g. a 5/3 split where
4/4 was achievable, past parity. Balance the combined count in one pass; disk-level
data/parity anti-affinity stays in pickBestDiskOnNode.
* ec.balance: don't let the imbalance threshold skip an over-parity machine
The within-rack spread gated on relative skew ((max-min)/avg > threshold), so a
worker threshold of 0.5 skipped an exactly-50%-skewed layout like 5/4/3 for a 10+4
volume, leaving 5 shards -- past parity -- on one machine. The even cap
(ceil(shards/groups)) is the real bound and the move loop already sheds only what
exceeds it, so drop the threshold gate from the within-rack phase (machine and node):
a balanced rack stays a no-op while any over-cap machine is always fixed.
* ec.balance: keep the imbalance threshold for the node fallback
Dropping the threshold from the whole within-rack phase made the node fallback too
eager: it runs only when machine fault tolerance is unachievable, so it is cosmetic
load distribution that should defer to the global utilization phase. Without the
gate it would, for a one-server-per-host 6/4 split at threshold 0.5, schedule a count
move that worsens utilization balance. Restore the threshold there; machine spreading
keeps bypassing it, since that bound is durability, not cosmetic skew.
* fix(s3api): require space in v2 auth prefix
Problem: Signature V2 Authorization headers with a malformed algorithm token such as AWSX... are accepted as if they were AWS ... headers.
Root cause: validateV2AuthHeader checks HasPrefix("AWS") but then slices past an assumed trailing space, so an extra character after AWS is skipped and the rest is parsed as credentials.
Fix: Require the Authorization header to start with the exact AWS plus space prefix before parsing fields.
Reproduction: go test ./weed/s3api -run 'TestValidateV2AuthHeader/algorithm_prefix_without_space|TestDoesSignV2Match/malformed_auth_-_no_space_after_AWS' -count=1 fails before the fix because AWSXAKIA... is accepted.
Validation: go test ./weed/s3api -run 'TestValidateV2AuthHeader/algorithm_prefix_without_space|TestDoesSignV2Match/malformed_auth_-_no_space_after_AWS' -count=1; go test ./weed/s3api -count=1; git diff --check; git diff --cached --check
* Update weed/s3api/auth_signature_v2.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Problem: Multipart uploads where the first part was a form field and a later part contained the file used the first part's Content-Type for the file metadata.
Root cause: After finding a later part with a filename, parseUpload copied data and MD5 from part2 but read Content-Type from the original part variable.
Fix: Read Content-Type from the discovered file part.
Reproduction: go test ./weed/storage/needle -run TestParseUploadUsesDiscoveredFilePartContentType -count=1 failed before the fix because the parsed MIME type was text/plain instead of application/x-seaweed-test.
Validation: go test ./weed/storage/needle -run TestParseUploadUsesDiscoveredFilePartContentType -count=1; go test ./weed/storage/needle -count=1; git diff --check; git diff --cached --check
VidCache.cache was a []VidInfo indexed directly by volume id, so caching
one volume with a large id grew the backing array to that many entries
(each 48 bytes), allocating a zeroed slot for every unused id below it. A
single id of 32M cost ~1.5GB resident, plus geometric realloc churn as the
append loop doubled the array.
Use map[uint32]VidInfo so memory scales with the number of volumes actually
cached rather than the largest id seen. Parse ids with ParseUint(.,32) so
values outside the uint32 volume-id range are rejected instead of silently
wrapping into a key.
* s3: set outbound bind IP before the first filer dial
Standalone weed s3 dialed the filer for GetFilerConfiguration before
SetOutboundLocalIP ran, so that gRPC conn was created with the stock
dialer and no source address. gRPC caches conns by address and reuses
the original dialer on reconnect, so the s3->filer connection kept
leaving from the OS-chosen source for the life of the process even
after the bind IP was set a moment later.
* grpc: install the outbound-bind dialer unconditionally
The dialer was installed only when OutboundLocalAddr was already set at
GrpcDial time, baking the source-address decision into the cached conn,
so a conn dialed before the bind IP was configured never bound.
Install the context dialer always and decide per dial: bind through
OutboundDialContext once a source is set, otherwise fall back to the
stock net.Dialer so default deployments keep gRPC's dial timeout and
keepalive behavior. The bind now applies on the next reconnect
regardless of ordering, matching the HTTP transport's unconditional
DialContext.
- Different-users bucket test: use getNewBucketName() so the bucket carries the
tracked prefix and run id and gets swept if the test leaks, instead of an
untracked name.
- Makefile: clarify that '.' matches the opt-in stress tests but they self-skip
without ENABLE_STRESS_TESTS, so they don't execute in the default run.
- Versioned list test: guard the Object.Size dereference with require.NotNil.
A suspended-versioning DELETE was recorded with createDeleteMarker, which mints a
fresh real version id each time, so repeated suspended deletes piled up delete
markers instead of overwriting a single null marker as S3 specifies. Record the
suspended delete as a 'null' marker with a fixed file name (v_null) and point the
latest-version pointer at it explicitly; putSuspendedVersioningObject's existing
null-version cleanup removes it on the next suspended PUT, so the object undeletes
cleanly and at most one null marker exists. Enabled-versioning deletes are
unchanged (still distinct historical markers).
Update TestSuspendedVersioningDeleteBehavior to the AWS-correct counts: one null
marker after a suspended delete, and the null marker plus one real marker after a
re-enabled delete.
* test(s3): correct bucket-recreate expectations and cover the different-owner case
A same-owner CreateBucket on an existing bucket returns BucketAlreadyOwnedByYou
(idempotent recreate); the suite expected BucketAlreadyExists, which only applies
when the name is owned by someone else. Fix the same-owner cases (plain and
Object-Lock) and implement the previously-skipped different-owner test, which now
exercises the BucketAlreadyExists path via a second identity.
* test(s3): assert the deletion invariant for suspended-versioning delete
A suspended-versioning DELETE removes the null version and records a delete marker
so the object reads as deleted; the test expected no marker, which would let an
older version resurface. Assert that a marker is recorded (and read DeleteMarker
through aws.ToBool) rather than an exact count, so it holds whether or not the
suspended-marker id/dedup is later collapsed to AWS's single null marker.
* test(s3): run the whole versioning suite by default
TEST_PATTERN was TestVersioning, which left bucket-creation, suspended-delete and
directory/version-listing tests ungated. Default to '.' so every test runs; opt-in
stress tests self-skip without ENABLE_STRESS_TESTS and keep their own targets.
TestVersionedObjectListBehavior compared int64 against listedObject.Size,
which is *int64, so the assertion always failed on a type mismatch once
reached. Dereference it (and in the log line).
ListObjectVersions gated explicit directory objects on Mime ==
FolderMimeType, but an SDK PutObject of "dir/" carries a default
Content-Type (e.g. application/octet-stream), so those directory keys
were dropped from the version listing while ListObjectsV2 - which keys
off IsDirectoryKeyObject (any non-empty mime) - still showed them. Use
the same IsDirectoryKeyObject check so the two listings agree.
The directory test's storage-class assertion compared an ObjectStorageClass
constant against ObjectVersion.StorageClass (ObjectVersionStorageClass);
the values matched but the SDK enum types did not, so it only surfaced
once the directories started appearing. Use the matching constant.
* s3: rescan .versions when the cached latest pointer is missing on a list
ListObjectsV2 resolves each versioned object's current version from the
latest-version pointer cached on the .versions directory entry. When that
pointer is absent on the filer serving the list, the object was dropped
from the listing. Fall back to a read-only rescan of .versions/ to pick
the newest version - the version files are present locally even when the
cached pointer is not - so the object still lists. This mirrors the read
path's recoverLatestVersionWithoutPointer; the scan loop is shared.
Read-only by design: a list can touch many objects, so it does not persist
a pointer.
* s3: copy scanned Extended before stamping the version id
applyRecomputeLatest wrote the .versions latest-version pointer and the
demoted prior version's stamp through UpdateEntry without a following
NotifyUpdateEvent, so neither change entered the metadata log. Across
filers the pointer then lived only on whichever filer ran the mutation,
and ListObjects served by any other filer dropped those objects from a
versioned bucket. Emit the events the way PATCH_EXTENDED already does,
keeping a pre-update image for the notification diff.
Stamping an Expiration.Days rule as a volume TTL at write time bakes an
irreversible TTL into the object: removing or lengthening the rule later
can't un-expire it, unlike worker-driven expiration. The metadata-only
delete it enables also skips per-chunk DeleteFile, so dead bytes linger in
a not-yet-expired TTL volume with no deleted-byte accounting until the
whole volume ages out.
Gate the resolver on a per-bucket flag, off by default; toggle with the
s3.bucket.lifecycle.fastpath shell command. Default writes take the worker
path: real deletes that honor current policy and let vacuum reclaim space.
* fix(helm): deduplicate all-in-one extra environment variables
The all-in-one Deployment looped global.seaweedfs.extraEnvironmentVars and
allInOne.extraEnvironmentVars in two separate ranges, so any key present in
both maps was emitted as two env entries with conflicting values. It also
computed a merged map for the cluster-default lookup but never used it for
the env loop.
Use the existing seaweedfs.mergeExtraEnvironmentVars helper (as the filer,
master and s3 templates already do) so a key set in both maps renders once
with the component value taking precedence, and add a chart-CI render
assertion covering it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
* ci(helm): drop checkmark glyphs from chart test output
---------
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(helm): suspend bucket versioning for YAML bool false
createBuckets[].versioning accepts both a YAML bool and a string. The
string branch maps "false"/"disable"/"suspended" to Suspended, but the
bool branch only handled true (Enabled) and left false as a silent no-op.
The same logical value therefore behaved differently depending on its
YAML type: `versioning: false` did nothing while `versioning: "false"`
suspended the bucket.
Mirror the string behaviour in the bool branch so bool false suspends the
bucket, and add a chart-CI render assertion covering it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
* ci(helm): trim versioning regression-test comment
* chart: document bool false for createBuckets versioning
---------
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* ip.bind: bind outbound connections to the configured address
-ip.bind only governed listeners; outbound gRPC and HTTP connections let
the OS pick the source IP, which may not even be able to reach the
target. Mirror the bind address into a process-global source address and
apply it to outbound TCP dials: the gRPC context dialer, the per-client
HTTP transports, and the default transport. Loopback targets and unix
sockets keep the OS-chosen source so same-host traffic still works.
* ip.bind: first-write-wins source IP, skip on address-family mismatch
Make SetOutboundLocalIP first-write-wins so a `weed server` component's own
bind setting (run in its goroutine) can't clobber the process-wide source
address the top-level -ip.bind already established for the other components.
Skip source binding when the target is a literal IP of a different family
than the bind address, since forcing a mismatched source fails the dial.
* fix(s3api): standardize ETag calculation across S3 API handlers
* s3: make copyEntryETag nil-safe
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* master: grow rack-spanning volumes once per DC, capped at copy_N
The periodic rack-aware growth scan grew once per rack. For rack-spanning
replication (DiffRackCount > 0) a single logical volume already covers every
rack the placement needs, so a crowded volume made every rack report
should-grow and the scan created racks×step too many volumes: with "010"
across two racks that is 2 racks x step 2 = 4 logical (8 physical) volumes.
Plan one DC-wide grow for rack-spanning replication, and cap the per-event
step at master.volume_growth.copy_N so lowering it reduces periodic growth.
* master: distribute lastGrowCount evenly across uneven DCs
The non-rack-spanning grow divisor used the current DC's rack count, so DCs
with different rack counts each over-grew. Sum every rack up front and divide
lastGrowCount by that global count instead.
* security: reload JWT signing keys on SIGHUP
Signing keys were read once in the server constructors and never
refreshed. After a key rotation (Secret update, divergent reads) the
in-memory key stayed stale and every request kept failing "wrong jwt"
until the affected process was restarted.
Add Guard.UpdateSigningKeys and call it from the master, volume and
filer reload paths and the s3 reload hook, next to the existing
whitelist refresh. Make the global chunk-read JWT cache reloadable via
an atomic swap, and register the master's Reload with grace.OnReload --
it was never wired, so the master ignored SIGHUP entirely.
Mirror the same refresh in the Rust volume server's SIGHUP handler.
* security: swap signing keys behind an atomic pointer
Addresses review feedback on the in-place key swap: SigningKey is a
[]byte, so reassigning the Guard fields while a request handler reads
them is a data race that can tear the multi-word slice header and read
out of bounds.
Hold the four signing-key fields in an immutable signingConfig snapshot
behind atomic.Pointer; UpdateSigningKeys swaps the whole pointer, so a
reader sees either the old keys or the new ones. Reads go through new
SigningKey/ExpiresAfterSec/ReadSigningKey/ReadExpiresAfterSec accessors.
The Rust guard is already safe: every read and the SIGHUP write go
through the shared RwLock<Guard>.
* security: fold whitelist + auth state into the atomic snapshot
Review follow-up. UpdateSigningKeys still wrote isWriteActive while the
request path read it (and the whitelist maps) unsynchronized, so a SIGHUP
under load could expose an inconsistent mix of activation bits and
whitelist contents.
Move all hot-reloadable Guard state -- keys, expirations, whitelist, and
the activation flags -- into a single immutable guardState swapped behind
one atomic.Pointer. The Update* methods take a small mutex to serialize
the read-modify-write; readers stay lock-free. The concurrency test now
also rotates the whitelist and probes IsWhiteListed under -race.
Also read each signing key once per branch in the volume/filer JWT auth
checks, so a reload landing mid-check can't take the allow-fast-path
after auth was enabled or verify against a different key than the branch
saw.
* fix(master): advance maxVolumeId when registering EC shards
After EC encoding the original normal volume is deleted, so a
high-numbered volume can exist only as EC shards. Only regular volumes
advanced maxVolumeId (Disk.doAddOrUpdateVolume), so a master that
rebuilt its state from heartbeats (raft state not resumed) undercounted
the max and NextVolumeId could re-issue an id that EC shards still
occupy. A new volume then gets created on top of the EC volume id; new
writes land on it, but reads route to the old EC shards whose .ecx never
held the new needle, returning 404 and corrupting that object.
Advance maxVolumeId when EC shards are registered, mirroring the
regular-volume path. RegisterEcShards is the chokepoint both the full
and incremental heartbeat sync paths funnel through.
* test: cover incremental heartbeat path for EC maxVolumeId
Both SyncDataNodeEcShards and IncrementalSyncDataNodeEcShards funnel
through RegisterEcShards; assert the invariant on the incremental path
too.
* s3: return BucketAlreadyOwnedByYou when recreating your own bucket
PutBucket returned BucketAlreadyExists for every existing bucket, even
when the caller already owns it, so idempotent re-creation (e.g. a
container that creates its bucket on startup) couldn't tell "someone
else took the name" from "it's already mine".
Recreating a bucket you own now returns BucketAlreadyOwnedByYou, unless
the request conflicts with the existing bucket: a different Object Lock
setting, or an ACL on the request or the existing bucket. To detect the
latter, a requested non-default canned/grant ACL is now persisted on
creation instead of being dropped.
* s3: fail PutBucket when the existing bucket's config can't be read
When a bucket already exists, an unreadable config left the recreate
defaulting to BucketAlreadyOwnedByYou, masking the backend error and
possibly accepting a conflicting recreate (Object Lock / ACL unknown).
Surface the read error instead.
* s3: return the stored bucket ACL from GetBucketAcl
GetBucketAcl always returned the owner's default full-control grant and
ignored any stored ACL, so a bucket created with a canned ACL or one set
via PutBucketAcl never read back correctly. Decode the stored grants
instead, sharing one grants-to-XML helper with the object ACL handler.
The shared helper also emits each grantee's real xsi:type (e.g. Group for
public-read) instead of a hardcoded CanonicalUser, so group grants read
back correctly for both bucket and object ACLs.
* s3: resolve the right already-exists error on the concurrent-create race
When two requests create the same bucket at once, the loser's mkdir
fails and the handler fell back to a flat BucketAlreadyExists, bypassing
the same-owner idempotency check. Route both the pre-check and the race
fallback through one existingBucketError helper so a same-owner recreate
still gets BucketAlreadyOwnedByYou.
* s3: record the bucket owner's account id at creation
setBucketOwner only stored the creating identity name, so the canonical
account id wasn't available later. Persist it under ExtAmzOwnerKey too,
the same field PutBucketAcl writes, so the bucket owner can be reported
independently of whoever reads it.
* s3: report the bucket owner from GetBucketAcl, not the caller
GetBucketAcl built the ACL Owner from the caller's account header, so an
admin or cross-account read returned the wrong owner. Use the owner
persisted on the bucket, falling back to the caller only when none is
recorded.
* mysql: keep S3 list order byte-lexicographic regardless of name column collation
ORDER BY name and the name > ? pagination predicate follow the column
collation, so a case-insensitive filemeta.name (e.g. utf8mb3_general_ci)
returns S3 keys out of byte order and breaks clients that merge two sorted
listings.
Detect the live name collation at startup; only when it isn't binary, wrap
the list comparison, prefix, and ORDER BY in BINARY name so order and
pagination stay consistent. Correctly configured utf8mb4_bin tables keep
their indexed range scan unchanged, and the operator gets a warning to
convert the column.
* postgres: keep S3 list order byte-lexicographic regardless of name column collation
ORDER BY name and the name > $n pagination predicate follow the column or
database collation, so a locale-aware filemeta.name (e.g. the en_US.UTF-8
database default) returns S3 keys out of byte order and breaks clients that
merge two sorted listings.
Detect the live name collation at startup; only when it isn't byte-ordered,
wrap the list comparison, prefix, and ORDER BY in COLLATE "C" so order and
pagination stay consistent. A byte-ordered (C/POSIX/C.UTF-8) column keeps its
indexed range scan unchanged, and the operator gets a warning to declare the
column COLLATE "C".
* filer: stream persisted log files when serving metadata subscriptions
readFileEntries buffered every LogEntry of a whole log file into memory
before returning them one by one, making a subscription read O(entries in
one log file) instead of O(one entry). On a filer with large per-entry
metadata, many concurrent SubscribeMetadata streams each loading a full
log file exhausted memory.
Keep a current LogFileIterator and return one entry at a time, advancing
files as each is exhausted. The deleted-volume skip is preserved.
* filer: close the log file iterator on read errors too
A genuine read error returned early without closing the current
LogFileIterator, leaving its ChunkStreamReader alive until GC. Close on
every exit path and propagate only a real error.
* filer: close persisted-log iterators when a subscription stops early
The streaming iterator keeps a log file reader open across calls, so a
subscription that returns before EOF (early stop, cancellation) left the
reader alive until GC. Add idempotent Close on LogFileQueueIterator and
OrderedLogVisitor, and have ReadPersistedLogBuffer wait for the readahead
goroutine and close the visitor on the way out.
* fix: log meta backup offset errors
* fix: log meta backup offset errors
* fix: exit on meta backup offset errors
Exit with a non-zero status when the initial metadata backup offset cannot be persisted.
Classify offset-read failures during streaming so the backup process exits instead of retrying forever, allowing supervisors to restart and bootstrap from a missing checkpoint.
* meta backup: read offset in the loop, drop offset error type
Reading the saved offset inside the retry loop makes an offset read
failure a clean exit and a stream error a retry, without a typed error
to tell them apart. streamMetadataBackup now takes the start time.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: keep dynamic IAM live when -iam.config is set
-iam.config was treated like a static -config identity file: it set
useStaticConfig, which makes the filer metadata subscription skip
reloads. Identities and policies created at runtime (the IAM gRPC API)
then never took effect, so advanced IAM (OIDC/STS) and dynamic IAM were
mutually exclusive.
Gate useStaticConfig on whether inline identities were actually loaded.
An OIDC/STS-only config carries none, so it keeps the dynamic credential
store live; a -config identity file still freezes its identities as
before.
* s3: mark static identities on config reload too
A -config reload (grace.OnReload) re-reads the file, but only the startup
path marked its identities static, so identities added to the file and
reloaded were left unprotected from dynamic filer updates. Move the
marking into loadS3ApiConfigurationFromFile and make it additive and
scoped to the file's identities, so a reload protects newly added ones
without freezing dynamic filer-managed identities.
* s3: sync reloaded static identities into the credential manager
After marking a (re)loaded config file's identities static, push the
updated set into the credential manager so reloaded identities still
appear in listings and survive later dynamic merges. Centralize the sync
in loadS3ApiConfigurationFromFile and drop the now-redundant call in the
reload hook.
2026-06-03 23:28:25 -07:00
Fabian HardtGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Chris Lu
* sftpd: support SSH user certificates signed by a trusted CA
Adds a new "certificate" auth method to weed sftp. When enabled, the server
loads trusted CA public keys from -trustedUserCAKeysFile (OpenSSH
authorized_keys format, one or more keys) and accepts only ssh.Certificate
blobs of type UserCert on the public-key channel. Validation uses
ssh.CertChecker: CA signature, ValidAfter/ValidBefore, non-empty
ValidPrincipals and SSH login user must appear in ValidPrincipals. The
authenticated user must exist in the user store; home dir and permissions
resolve as before.
Behaviour mirrors MinIO's --sftp=trusted-user-ca-key and OpenSSH's
TrustedUserCAKeys: when certificate auth is active, plain (non-cert) public
keys are rejected even if "publickey" is also listed. Default authMethods
remain "password,publickey", so existing deployments are unaffected.
* Update weed/sftpd/auth/certificate.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* sftpd: address review feedback on certificate auth
- Pre-marshal trusted CA public keys in IsUserAuthority instead of
re-marshaling on every authentication attempt (gemini-code-assist).
- Differentiate user-not-found from underlying store errors via
errors.As(*user.UserNotFoundError) so backend/read failures are no
longer reported as bad credentials (coderabbitai).
- Fix the corresponding sanity check in the missing-file test to use
errors.As instead of errors.Is (UserNotFoundError has no Is method,
so the previous check never matched) (coderabbitai).
* sftpd: register trustedUserCAKeysFile flag in filer and server commands
The new field on SftpOptions is dereferenced unconditionally in
resolvePaths(), but only the standalone `weed sftp` command was wiring
its flag. `weed filer` and `weed server` both embed an SftpOptions value
and call resolvePaths() on it, so they hit a nil pointer dereference at
startup.
Register `-sftp.trustedUserCAKeysFile` in both commands and update the
-sftp.authMethods help text to mention the new "certificate" method.
Fixes the SFTP Integration Tests CI failure on this PR.
* helm: expose SFTP certificate auth in the SeaweedFS chart
Adds Helm-chart support for the new SSH user-certificate auth method:
- values.yaml (sftp:) gains `trustedUserCAKeys` (inline OpenSSH
authorized_keys-format CA public keys) and `existingCAKeysSecret`
(reference an externally managed Secret). Same pair added under
allInOne.sftp with a null default that falls back to the top-level
sftp.* setting.
- New template templates/sftp/sftp-ca-secret.yaml renders a
chart-managed Secret <release>-sftp-ca-secret with `ca_user.pub`,
but only when SFTP is enabled, "certificate" is in authMethods,
inline keys are provided, and no existingCAKeysSecret is set.
- templates/sftp/sftp-deployment.yaml and the all-in-one deployment
template add `-trustedUserCAKeysFile=/etc/sw/sftp_ca/ca_user.pub`
to the weed sftp command, mount the CA secret at /etc/sw/sftp_ca
and add the corresponding volume. All cert-auth bits are guarded
by `contains "certificate" authMethods` so existing users see no
change.
- authMethods help text updated to mention "certificate".
Verified end-to-end on a local k3d cluster: cert login succeeds,
plain-pubkey login is rejected with "public key without certificate
not allowed".
* helm: fail render when SFTP certificate auth lacks CA keys
When certificate is in authMethods but neither trustedUserCAKeys nor
existingCAKeysSecret is set, the deployment mounted a secret that the
chart never renders, leaving the pod stuck on a missing volume. Fail at
template time with a clear message instead.
* sftpd: fix stale auth-method list in SFTPServiceOptions comment
keyboard-interactive was never implemented; certificate is the new
supported method. Match the CLI help text.
* sftpd: test Manager wiring of certificate vs public-key channel
Cover the channel takeover at the Manager level: certificate auth
displaces plain public-key auth when both are enabled, public-key auth
stays put otherwise, and enabling certificate without a CA file errors.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: bound TraverseBfsMetadata memory by queuing directory paths
The BFS enqueued every entry, so it held the whole subtree in memory
including each file's chunk list. A filer serving a peer's first-time
bootstrap traversal of a large tree could exhaust memory and get killed.
Stream each entry as it is visited and queue only directory paths to
descend into. Memory is now bounded by the number of directories rather
than the entire tree, and the streamed output order is unchanged.
* filer: match excluded prefixes on path-component boundaries
Only treat an excluded prefix as a match when it ends at a path
boundary, so excluding /a/b does not also drop a sibling like /a/bc.
Short-circuit the trie walk on the first real match.
Blanking preferred kept route-by-key reads from dialing a flagged owner first,
but withFilerClientFailover always re-adds the current filer, so when the owner
is the gateway's current filer it stayed in the candidate list and got dialed
anyway. Treat a recently-unreachable filer as unhealthy in the health partition
so it is deferred to the last-resort tail instead of tried before healthy
replicas; preferred is still tried first, and a live owner is unaffected.
* s3: route object reads to the key's owner filer
Writes already route by key to the owner filer on the lock ring, where the
entry is created. Reads went to the gateway's local filer and treated its
NotFound as authoritative, so a GET on one gateway could miss an object
another gateway had just written until the filers' metadata replication
caught up.
Resolve an object's entry from the key's owner first, failing over to the
gateway's filer set only on transport errors. An owner NotFound stays
authoritative: no fan-out across filers, and no resurrecting a peer's
not-yet-replicated tombstone, so a delete routed to the owner is visible at
once and a genuine miss costs one lookup. Keys owned by the local filer are
unchanged. Objects written through the non-routed lock path land on a
gateway's local filer, so they can still read as absent on the owner until
they replicate.
withFilerClientFailover takes a preferred start filer; the object-entry
reads pass the owner, every other caller passes "" and keeps the
current-filer fast path.
* s3: consult the prior owner on a rebalance-window read miss
Owner-first reads route a key to its current ring owner. When a filer joins,
~1/N of keys reassign to it, and the new owner may not have replicated a
just-moved key yet, so an owner NotFound would surface a transient 404 for an
object that already exists elsewhere.
Retain the previous ring on the gateway's LockClient for a cooling-off window
(PriorOwnerForKey, mirroring the master's LockRing.PriorOwner) and, on the
owner's NotFound, probe the key's previous owner once before treating the miss
as final. The probe is scoped to keys whose ownership actually moved and only
within the window, so steady-state reads are untouched.
This trades the transient scale-up 404 for a transient stale read if a delete
routed to the new owner races the same window — the same authoritative-NotFound
tradeoff, narrowed to the rebalance.
* s3: try healthy filers before unhealthy ones on failover
The candidate list probed its first entry (usually the current filer)
unconditionally, so a health-flagged current filer cost a transport timeout on
every ordinary call before failover reached a replica. Partition candidates into
healthy and unhealthy, keep priority within each, and fall back to unhealthy
ones only when all healthy ones fail.
* reduce comments on the routed read and lock client paths
* s3: skip a recently-unreachable owner on route-by-key reads
The gateway's filer health tracking no-ops for an owner outside the static
-filer list, so during a sustained owner outage every route-by-key read
re-dials the dead owner before failing over. Flag an owner whose owner-first
read hit a transport error and skip it (read local-first) for a short TTL, so
reads pay one dead dial per TTL instead of one per request; the flag expires so
owner-first reads resume once the owner or the ring recovers.
* s3: always try the preferred owner first, health-order only the rest
The healthy/unhealthy partition also demoted a health-flagged preferred owner
behind healthy replicas, so a replica's authoritative NotFound could mask a
write that had only reached the owner — the read-after-write race this routing
exists to close. Pull preferred out of the partition and keep it first; the
recently-unreachable gate already steers reads away from a genuinely dead owner.
* fix(iam): return a valid user ARN from CreateUser and GetUser
The terraform aws provider 6.41 reads a user back after creating it and
blocks until GetUser returns a value that passes arn.IsARN. We only set
UserName, so the ARN was empty and apply hung until the 2m timeout.
Populate Arn (and Path) via a shared iam.NewUser helper in both the
embedded and standalone IAM handlers.
* fix(iam): use the userName parameter directly in NewUser
Drop the redundant local copy; the value parameter is already function-local.
* fix(iam): return full user objects with ARNs from GetGroup
GetGroup listed members with only UserName set. Build them via the shared
NewUser helper so group members carry a valid Arn and Path like the other
user responses, in both the embedded and standalone IAM handlers.
ReadFromBuffer and HasData() take the read lock separately, so a write
that lands between them can make a subscriber which just read a
momentarily empty buffer return ResumeFromDiskError even though the data
is now servable from memory. Re-read under a fresh lock and only bail
when the position is genuinely behind the in-memory window (flushed to
disk); otherwise loop back and read it.
* fix(iam): implement CreatePolicyVersion for managed policies
The AWS Terraform provider updates a managed policy in place via
CreatePolicyVersion, which returned 501 NotImplemented and broke
terraform apply on any policy change.
Implement CreatePolicyVersion (plus ListPolicyVersions, GetPolicyVersion
and DeletePolicyVersion) on both the standalone IAM server and the
embedded S3 IAM API. Managed policies keep a single current document, so
each is modeled as one default version "v1": CreatePolicyVersion replaces
the document, List/GetPolicyVersion expose it, and DeletePolicyVersion
rejects deleting the default. GetPolicy now reports DefaultVersionId so
the provider's read can fetch the document. The standalone path also
refreshes the cached Identity.Actions of every identity the policy is
attached to so the new document takes effect.
* fix(iam): reject CreatePolicyVersion unless SetAsDefault=true
With a single always-default managed-policy version, a request with
SetAsDefault=false (or omitted) would stage a non-default version on AWS
but here silently replaced the active document. Reject it on both the
standalone and embedded paths.
Isolate the new policy-version tests from the shared package fixtures so
they stay order-independent, and assert IsDefaultVersion on the response.
The username-isolation policy denied s3:ListBucket through an object-path
NotResource. ListBucket is bucket-level, so its resource ARN is the bucket
and never matches an object path: the Deny always fired and a user could
not list their own prefix. Scope the per-user List deny with a StringNotLike
s3:prefix condition instead, the same mechanism the matching Allow uses.
* fix(s3api): keep ListBucket resource ARN at bucket level
ListObjects with ?prefix= was denied for IAM users granted s3:ListBucket
on the bucket ARN. authRequestWithAuthType promotes the prefix into object
so the legacy CanDo path can honor prefix-scoped Action strings, and that
promoted object leaked into the policy resource ARN, producing
arn:aws:s3:::bucket/<prefix> which never matches a bucket-level statement.
Keep the resource bucket-level for List in the bucket-policy and
IAM-attached-policy evaluators; prefix scoping stays in the s3:prefix
Condition. The CanDo path is untouched.
* fix(s3api): resolve List action at bucket level when prefix is promoted
The IAM evaluator built a bucket-level resource ARN but still passed the
prefix-promoted object to ResolveS3Action, so listing with a prefix made
hasObject true and misresolved ListBucketVersions/ListBucketMultipartUploads
to ListBucket. Resolve the action against the same zeroed object, and trim
the resource-ARN comments.
Bulk DeleteObjects carries the keys in the request body, so the route Auth
middleware ran a single bucket-level check with object="", building the
resource ARN as arn:aws:s3:::<bucket>. That never matches an s3:DeleteObject
policy scoped to <bucket>/*, so the entire batch was denied even though the
single-key DELETE worked with the same credentials.
Defer authorization to the handler and check each key via AuthorizeBatchDeleteKey,
mirroring AuthorizeCopySource: a synthetic DELETE /<bucket>/<key> request resolves
s3:DeleteObject (or s3:DeleteObjectVersion when a versionId is given) against the
object ARN. Denied keys come back as per-key errors while authorized keys still
delete, matching AWS semantics.
* fix(mount): don't strand a directory cached-but-empty when an off-loop wipe races a rebuild
Idle eviction, kernel Forget, and the copy-range fallback cleared a
directory's cached entries directly, off the metaCache apply loop, after
resetting the cached flag in inodeToPath as a separate step. A concurrent
rebuild could publish a fresh listing (markCachedFn) in between, so the late
DeleteFolderChildren left the directory flagged cached over an empty store.
lookupEntry then returns an authoritative ENOENT and ReadDir returns nothing,
so every file in the directory disappears from the mount although it is still
present on the filer.
Route those wipes through a new apply-loop step that resets the flag and wipes
the store together, serialized with a build's markCachedFn, and skips a
directory while it is building.
* fix(mount): route the meta-event retry cleanup through the apply-loop purge
The subscription-retry callback wiped the mount root's cached children
directly off the apply loop and reset the cache flags as a separate step — the
same pattern that can leave a concurrently-rebuilding root cached-but-empty.
Invalidate all flags (safe on its own, it never deletes entries) then purge the
root's children through the apply loop.
* [CheckDisk][GRPC]: implement MVP for disk health detection, added timeout for new grpc connections
* fix(volume): build disk health check on every platform
setDiskStatus only existed behind the statfs build tag, so disk.go failed
to compile on windows, openbsd, solaris, netbsd and plan9. Move the timeout
wrapper and failure tracking into the shared disk.go and have each platform's
fillInDiskStatus return an error, so every platform gets the same protection
from a stuck filesystem.
Also restore the uint64(fs.Bavail) cast: Bavail is int64 on freebsd, so the
unguarded multiply broke the freebsd build.
* fix(volume): keep one outstanding statfs probe per disk
A stuck statfs used to leave isChecking cleared by the timeout path, so the
next check spawned another goroutine while the previous one was still blocked
in the syscall, leaking one goroutine per minute on a hung disk. Clear the
flag only when statfs returns and treat an overlapping check as a failure, so
a hung filesystem keeps a single outstanding probe and still gets reported.
* fix(volume): assume disk available until the first health check
isDiskAvailable defaulted to false, and CollectHeartbeat skips locations that
are not available. A freshly started volume server would therefore omit every
volume from its first heartbeats until the async CheckDiskSpace ran, so the
master could briefly treat all of them as missing.
* fix(volume): label the disk error metric by data directory
The new gauge tagged the series with IdxDirectory while every neighbouring
resource gauge uses Directory, so the error series would not line up with them
in dashboards. Also log the underlying error instead of a generic message.
* test(volume): cover disk health success and repeated-failure paths
* fix(volume): make a healthy disk the zero-value default
Track the disk as isDiskUnavailable instead of isDiskAvailable so the safe
state is the zero value, matching isDiskSpaceLow. CollectHeartbeat only skips a
location once a check has actively marked it unavailable, so any DiskLocation
built without running CheckDiskSpace (tests, future call sites) still reports
its volumes instead of silently dropping them.
* feat(disk): detect degraded disks using IO latency probes
* feat(stats): introduce configurable disk I/O health probe with EWMA-based latency detection
* feat(disk): replace EWMA with sliding window algorithm for disk health detection and added user-friendly options
* feat(disk): improve disk health probing and recovery
* feat(volume): configure disk health checks via volume.toml
* fix(volume): Remove disk IO probe CLI options
---------
Co-authored-by: ptukha <ptukha@tochka.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Explain:
- problem: Delete and DeleteProxied could panic on malformed URLs when a JWT was provided.
- root cause: maybeAddAuth was called before checking the error returned by http.NewRequest, so req could be nil.
- fix: return the request construction error before adding the Authorization header.
- validation: go test ./weed/util/http -run 'TestDelete(ReturnsInvalidRequestErrorBeforeAddingAuth|ProxiedReturnsInvalidRequestErrorBeforeAddingAuth)' -count=1; git diff --check
* fix(seaweed-volume): bound request body and stored-content expansion to prevent OOM
The Rust volume server buffered the entire upload body with
to_bytes(usize::MAX) and only checked the file-size limit afterward, so a
single large upload — or many concurrent uploads, since the in-flight byte
throttle defaults to 0 (unlimited) — could exhaust memory and get the process
OOM-killed under load. The read path had two more single-request OOM vectors:
`vec![0u8; manifest.size]` allocated from an attacker-controlled chunk-manifest
size, and gzip decompression was unbounded (gzip bomb).
- Bound the upload body read by file_size_limit_bytes (plus a margin for
multipart framing), mirroring Go's io.LimitReader(sizeLimit+1), and reject
oversize before the whole body is buffered.
- Validate manifest.size (reject negative / oversized) before allocating.
- Cap gzip output in maybe_decompress_gzip and route the inline GzDecoder sites
through it.
* fix(seaweed-volume): address review - chunk offset, 32-bit cast, decompress errors
- Validate chunk.offset before indexing in chunk-manifest expansion: a negative
offset wrapped to a huge usize and underflowed `end - offset` (panic from a
crafted manifest). Reject negative, skip out-of-range, use saturating math.
- Use usize::try_from for the upload body limit instead of `as usize`, so a
>usize::MAX file_size_limit on 32-bit caps at usize::MAX rather than silently
truncating to a tiny value.
- maybe_decompress_gzip now returns Result<_, GunzipError> distinguishing a
decode failure (callers fall back to raw bytes, as before) from hitting the
size cap (TooLarge), which now returns 413 instead of silently serving the
still-compressed bytes.
* fix(seaweed-volume): inflate manifest chunks into the result window to cap peak memory
The chunk-manifest expansion still doubled memory: `result` was already allocated
at manifest.size (<=2 GiB) and each compressed chunk was inflated into a separate
Vec (also up to 2 GiB), so a single request could peak near 4 GiB.
Decompress compressed chunks directly into their result[offset..] window (bounded
by the remaining space) so a chunk never allocates a second large buffer; peak
stays at ~manifest.size. Bytes past the window are dropped (matching the prior
truncation), and a fully-undecodable chunk still falls back to its raw bytes.
* fix(seaweed-volume): fall back to raw chunk bytes on any decode failure
Per review: the gzip fallback must run on any decode error, not only when no
bytes were decoded. Clear the partially-written output and copy the chunk's raw
bytes (truncated to the window), restoring the prior decode-failure behavior.
* docker: cross-compile the Go binary instead of emulating it under QEMU
The builder stage ran as the target platform, so arm64/arm/386 images
emulated the whole Go compile (and the full git clone) under QEMU. The
binary is CGO-free, so pin the builder to $BUILDPLATFORM and cross-compile
with GOOS/GOARCH (GOARM for v7), keeping every target's compile native.
* ci: build all release container variants in parallel
The build matrix throttled to two variants at a time on a stale rate-limit
worry. Pulls go through mirror.gcr.io and pushes target GHCR only, so the
five variants can all build at once.
* ci: copy each variant to Docker Hub from its build job
The separate copy-to-dockerhub job waited on the whole build matrix before
any GHCR -> Docker Hub copy could start. Move the crane copy into the build
job so each variant copies as soon as it is built, overlapping with the
others still compiling. tag-latest and helm-release now depend on build.
GetObject on a versioned object returned NoSuchKey forever when the
.versions directory existed but carried no latest-version pointer (empty
Extended metadata) while real version files remained inside it. The
self-heal path only fired for a dangling pointer (present but referencing
a missing file), not an absent one, so doGetLatestObjectVersion fell
straight through and errored on every read.
- doGetLatestObjectVersion now calls recoverLatestVersionWithoutPointer
when the pointer is missing or empty. An absent pointer is the legitimate
signal that a pre-versioning or suspended-versioning "null" object is
current, so that object wins; only when it is absent do we rescan
.versions/ and rebuild the pointer from the version files present.
Transient rescan failures propagate instead of being masked as NotFound.
- selectLatestVersion derives the version id from the v_<versionId> file
name when the Seaweed-X-Amz-Version-Id attribute is absent, so version
files written outside the normal versioned-PUT path (replicated or
restored entries) are still promotable. The orphan diagnostic uses the
same detection so an entry can't be both promoted and counted an orphan.
* fix(ec): pass per-volume data-shard count to the parity-shard split
ShardsInfo.DeleteParityShards/MinusParityShards looped ids 10..13, assuming
the fixed 10+4 layout. For a non-default ratio this splits data vs parity
wrong — a wide ratio (12+4, 16+6) drops real data ids >= 10, which breaks
ec.decode. They now take a dataShards argument (<= 0 falls back to
DataShardsCount) and clear ids dataShards..MaxShardCount. ec.decode threads
the data-shard count from collectEcNodeShardsInfo to both split call sites,
and admin LogicalSize passes DataShardsCount.
Also: EC cleanup now sets an explicit per-disk storage impact
(-len(ShardIds)) instead of falling back to the TotalShardsCount constant,
so freed-capacity accounting matches the shards actually removed.
OSS is always 10+4, so behavior is unchanged here; this keeps the split
ratio-correct and the API aligned with the enterprise per-volume override.
Adds parity-split ratio tests.
* ec: clear parity shards in one locked pass
Address review: DeleteParityShards looped si.Delete, taking the lock once per
id. shards is sorted by Id and shardBits is a bitmap, so mask off the high
bits and truncate the sorted slice at the first parity id (binary search) under
a single lock. Preserves the dataShards<=0 -> DataShardsCount default.
* fix(ec): resolve EC data-shard count from the volume's .vif on reboot
A volume server never loads a cluster EC config into memory, so startup
decisions that assumed 10 data shards mishandled volumes whose .vif
records a different ratio:
- validateEcVolume sized the expected shard against 10 data shards and
required >=10 local shards, so a volume with a non-default ratio and a
coexisting .dat could be wiped on reboot. Read the ratio from the .vif.
- pruneIncompleteEcWithSiblingDat used the hardcoded 10-shard threshold,
so a full data set for a non-default ratio with a healthy sibling .dat
was wiped as a partial leftover. Use the EcVolume's .vif-derived ratio.
Behavior is unchanged for the standard 10+4 layout (the .vif resolves to
10). Adds storage-level reboot tests.
* ec: avoid per-call allocations in ecDataShardsFromVif
Address review: the helper runs once per EC volume at startup. Replace the
slice+map dedup of the two dirs with direct conditional checks via a small
ecDataShardsFromVifDir helper, eliminating the heap allocations and GC
pressure when loading many volumes.
A Canceled/DeadlineExceeded from the caller's per-request context was
treated like a dead channel: it closed the shared cached ClientConn and
cancelled every other in-flight RPC on it with "the client connection is
closing". Under a burst of concurrent chunk assigns (e.g. a large S3
multipart upload) one slow assign hitting its 10s attempt timeout could
poison the connection for all the rest, cascading into a flood of 500s.
Thread the caller's context into shouldInvalidateConnection and only
invalidate on Canceled/DeadlineExceeded while that context is still live,
which isolates the genuine stale-channel signal (a peer restart behind a
k8s Service VIP). To carry the context, add a ctx parameter to the
existing WithGrpcClient, WithMasterClient, and WithMasterServerClient; the
master assign and volume-lookup paths pass their per-attempt context and
every other caller passes context.Background().
makeupDiff replays post-snapshot changes onto the compacted volume. For a
replayed deletion it appended a tombstone to the new .dat but recorded the
.idx entry with offset 0. When that deletion is the last replayed change the
tombstone lands at the .dat tail, and the post-commit integrity check skips
offset-0 entries, so it sees 32 trailing bytes it can't account for and flips
the volume read-only, reloading it as a SortedFileNeedleMap instead of the
writable map.
Record the tombstone's real .dat offset, matching the normal delete path; the
needle map still treats it as deleted off the negative size, so lookups are
unchanged. Mirror the same fix into the Rust volume server.
* s3: auto-enforce bucket quota read-only both ways
Quota read-only only ever flipped when an admin re-ran
s3.bucket.quota.enforce, so a bucket that went over quota stayed
read-only forever even after usage dropped back under.
Fold enforcement into the per-minute, leader-locked bucket-size loop
the s3 gateway already runs for metrics: it now flips each bucket's
read-only flag to match its quota in both directions, rewriting
filer.conf only when a flag actually changes. The set/clear decision
lives in one shared FilerConf.ApplyBucketQuotaReadOnly helper so the
shell command and the gateway can't drift.
* only manage read-only when a quota is set, never clobber manual locks
* trim comments
When the destination's stored mtime is newer than the incoming source version, UpdateEntry skips the update (last-writer-wins). A copy left truncated by an earlier failed replication trips this: the source kept the file's original mtime while the partial copy was written recently, so it looks "newer" and is never corrected. When the destination is strictly shorter than the source, re-replicate the full source content and replace the chunk list instead of skipping. Same shorter-than-source bypass for CreateEntry.
* s3api: use getEtagFromEntry for multipart part ETag to prefer Extended metadata
* s3api: add tests for getEtagFromEntry Extended ETag preference in multipart upload
* s3api: avoid double-quoting ETags in ListParts output
* s3api: add docstring for filer_multipart_etag_test.go
* fix(http): handle invalid gzip stream errors
Explain:
- problem: ReadUrlAsStream could panic when a response claimed gzip encoding but the body was not a valid gzip stream.
- root cause: the gzip reader error was ignored and a nil reader was deferred and read from.
- fix: return the gzip.NewReader error before registering Close or reading.
- validation: go test ./weed/util/http -run TestReadUrlAsStreamReturnsGzipReaderError -count=1; git diff --check.
* test: avoid closing shared global HTTP client in unit test
* filer: name the read-only path in the write rejection
The write path rejected creates under a read-only rule with a bare
"read only", giving no hint which path was locked or why. Wrap the
error with the matched location prefix and a quota hint so a FUSE
mkdir or S3 put points straight at the offending bucket.
* return the read-only reason over HTTP and drop any query string from the fallback prefix
A destination volume server that hits its idle deadline while reading a large upload body under load returns 400 "read tcp ...: i/o timeout". fetchAndWrite retried that on the flat ~1s RetryUntil backoff, hammering the already-overloaded destination. Route i/o timeout, connection reset, broken pipe and net.Error timeouts through the same escalating 10s-2min backoff already used for EOF so it can recover.
shell: only skip system-log subtree in fs.meta.save, not fsck/verify
The SystemLogDir skip lived in the shared BFS traversal, so volume.fsck
built its in-use set without the /topic/.system/log chunks and flagged
every referenced log needle as orphan. -reallyDeleteFromVolume would then
delete live log data and leave dangling filer entries. Gate the skip
behind a flag that only fs.meta.save sets.
* fix(ec_bitrot): cap sidecar block_size in ValidateBitrotManifest
A sidecar loaded from disk (or supplied via a backfill/peer RPC) could carry a
huge power-of-two block_size that passed validation, then force a multi-GiB
scratch-buffer allocation in scrub/verify. Add a shared MaxBitrotBlockSize
(64 MiB) constant, enforce it as an upper bound in isPow2MultipleOf1MiB, and
derive the volume flag cap from the same constant so they cannot drift.
* fix(ec_bitrot): don't destroy a valid destination sidecar on an optional copy
writeToFile opened the destination with O_TRUNC before knowing whether the
source had the file, so an optional copy (ignoreSourceFileNotFound) from a source
that lacks the .ecsum truncated and then removed a valid pre-existing destination
sidecar. Stage the optional copy into a temp sibling and commit it with an atomic
rename only when the source actually delivered the file; a missing source is now
a no-op. Mandatory copies keep their in-place behavior.
* ec: add EC bitrot checksum protobuf
EcBitrotProtection/EcShardChecksums/ChecksumAlgorithm sidecar messages,
copy_ecsum_file and unsafe_ignore_sidecar fields, and a CHECKSUM scrub mode.
* ec: bitrot checksum sidecar format, validation, and per-volume load
Per-shard CRC32C block checksums in an optional <base>.ecsum sidecar with a
self-integrity header; validation, rolling builder, backfill primitive, and
EcVolume load on mount + removal on destroy.
* ec: capture per-shard checksums at encode; verify-and-exclude on rebuild
WriteEcFilesWithContext returns the protection computed inline during encoding.
generateMissingEcFiles verifies present inputs against the sidecar, excludes
corrupt ones, regenerates in place, and re-verifies; fail-closed unless
unsafe_ignore_sidecar, removing all generated outputs on failure.
* ec: read-only checksum scrub with Reed-Solomon arbiter
ChecksumScrub verifies each local shard against the sidecar and reconstructs
flagged shards from the clean shards so stale-sidecar false positives are not
reported. Wired to the gRPC CHECKSUM mode and ec.scrub -mode checksum.
* ec: server-side bitrot sidecar write, copy, cleanup, and opportunistic backfill
Write .ecsum at fresh encode; propagate it with copy_ecsum_file (tolerant);
remove it on full delete and decode; rebuild honors unsafe_ignore_sidecar and
opportunistically backfills a sidecar when all shards are reachable.
* ec: volume server bitrot config flags
-ec.bitrotChecksum (default on) and -ec.bitrotBlockSizeMB (default 16).
* fix(ec_bitrot): bound -ec.bitrotBlockSizeMB before the int64 multiply
Validate the MiB value is in [1, 1024] before multiplying by 1 MiB, so a huge
flag value cannot overflow int64 and slip past the power-of-two check, and a
block size cannot collapse a sidecar to a few oversized blocks.
* fix(ec_bitrot): distribute the .ecsum sidecar from the worker encode path
The worker EC encode wrote the generation-0 sidecar locally but never added it
to shardFiles, so DistributeEcShards never shipped it and the distributed
holders came up unprotected. Append it to shardFiles and map the ecsum shard
type to its extension in the sender so it travels with the shards.
* fix(ec_bitrot): remove orphaned sidecars when the generation is gone
Gate sidecar removal on existingShardCount==0 alone rather than also requiring a
stray .ecx. A sidecar whose shards have all been deleted is orphaned and must be
removed even when no .ecx remains, or it leaks. .ecx/.ecj/.vif removal stays
gated on hasEcxFile as before.
* fix(ec_bitrot): do not fold checksum blocks scanned into TotalFiles
ChecksumScrub's first return is blocks scanned, not files. Discard it so the
scrub response's TotalFiles (a needle/file count) is not inflated by the block
count for CHECKSUM mode.
* test(ec_bitrot): clean up generated .ecsum sidecars in removeGeneratedFiles
* fix(ec_bitrot): reject an oversized sidecar payload before the uint32 cast
The header stores payload_len as a uint32; bound the payload before the
conversion so a pathological manifest cannot truncate the length field and
corrupt the sidecar. A real manifest is a few KB, so this never trips.
* fix(ec_bitrot): cap -ec.bitrotBlockSizeMB at 64 MiB
The block size becomes the per-shard scratch buffer the scrub/backfill path
allocates, so an over-large value (e.g. 1 GiB) is a memory hazard per concurrent
scrub worker. Lower the upper bound from 1024 to 64 MiB.
* fix(ec_bitrot): add -ecUnsafeIgnoreSidecar to weed tool fix -ecx
The -ecx recovery path reconstructs missing shards via RebuildEcFilesWithContext,
which fails closed on a malformed/stale .ecsum. Without an override flag an
operator could not complete the rebuild without manually deleting the sidecar.
Expose -ecUnsafeIgnoreSidecar (default false) and thread it through.
* fix(ec_bitrot): bound sidecar payload with a direct int constant; drop readFull
Guard len(payload) against a plain int constant (1 GiB) before the allocation
instead of a uint64 MaxUint32 compare, so the allocation-size value is provably
bounded (clears the CodeQL overflow alert) and the math import is no longer
needed. Inline os.File.ReadAt with io.EOF handling in verifyShardFileBlocks and
remove the now-redundant readFull helper (os.File.ReadAt fills the slice or
errors).
* test(ec_bitrot): use slices.Contains instead of a hand-rolled containsU32
* refactor(ec): fold the EcFiles WithContext variants into the base functions
RebuildEcFiles now takes the *ECContext directly (nil => derive from .vif as
before) and WriteEcFiles takes it too (nil => default), removing the parallel
RebuildEcFilesWithContext / WriteEcFilesWithContext names. Callers that had an
explicit context drop the WithContext suffix; the default-context callers pass
nil. No behavior change.
* refactor(ec): pass BackgroundECContext instead of nil to Write/RebuildEcFiles
Add a non-nil BackgroundECContext placeholder (analogous to context.Background())
and have callers with no specific layout pass it instead of a nil *ECContext.
WriteEcFiles resolves a zero/background context to the default ratio and
RebuildEcFiles resolves it from the .vif, so behavior is unchanged.
* fix(ec_bitrot): make BackgroundECContext a func; RebuildEcFiles fails closed on bad .vif
- BackgroundECContext is now a function returning a fresh *ECContext, so callers
cannot mutate a shared singleton or race on it (and it mirrors context.Background,
which is also a function).
- RebuildEcFiles now propagates the MaybeLoadVolumeInfo error: a present-but-
unreadable .vif fails closed instead of silently rebuilding with the default
ratio (which would corrupt a custom-ratio volume). Pass an explicit ctx to override.
* shell: warn in volume.list when a volume id spans collections
A reused volume id, the result of the master handing out an id already
used by another collection (for example after losing its max-volume-id
counter on restart), makes collection.delete destroy the wrong
collection's data and makes any bare-id lookup, move, or vacuum
ambiguous. volume.list now scans the full topology and warns on ids
present in more than one collection so the clash is visible before any
destructive operation.
* volume.list: track duplicate ids lazily, sort with slices.Sort
Allocate the per-id collection set only on the first cross-collection clash
instead of one set per volume, so allocations scale with duplicates rather
than the volume count.
filemeta is the filer SQL store's default table name. A bucket of that
name passes VerifyS3BucketName but is rejected by the store's isValidBucket
guard on every operation, so it creates fine yet can't be deleted and wedges
fsck. Reject it at creation so both checks agree.
* s3: commit a versioned PutObject and its latest pointer in one transaction
A versioned PutObject wrote the version file and flipped the .versions
latest pointer in two separate routed transactions. Fold the
RECOMPUTE_LATEST into the version file's PUT so both commit atomically
under the object's per-path lock: the recompute, applied after the PUT in
the same transaction, scans the directory and sees the new version. A
crash can no longer leave the version present with a stale pointer.
putToFiler now takes a putFinalize describing the finalize step — routed
mutations folded into the PUT, or an afterCreate run under the object
write lock off the ring. Suspended-versioning keeps its afterCreate-only
form; multipart, copy, and delete-marker finalizes are unchanged.
* s3: trim verbose finalize comments
uploadFileGrpc passed SaveSmallInline with a 256 KiB limit, so uploads under
that size were written to entry.Content instead of a volume. The filer's own
upload path never inlines unless saveToFilerLimit is set (default 0), and the
S3 server shares that path. Drop the inline options so admin uploads always
land in volumes.
The viewer embedded images and PDFs through the download URL, which sent
Content-Disposition: attachment, so the browser downloaded them instead of
rendering. Add an inline mode to the download endpoint, limited to images and
PDFs so a hostile upload (HTML, SVG) can't run as same-origin script, set
X-Content-Type-Options: nosniff, and resolve the MIME the same way the viewer
does. The viewer now requests the inline URL.
* volume: don't drop the last writable slot on auto-sized disks
MaybeAdjustVolumeMax subtracted 1 from the per-disk slot count, so a disk
with room for exactly one volume (free between 1x and 2x the size limit)
reported 0 slots. The master then never grew a writable volume and every
assign drained its retry budget, so writes failed with context deadline
exceeded. Count the full volumes that actually fit, floored at one for an
auto-sized disk that has free space.
* mini: show disk and volume capacity in the startup banner
Print free space, volume size, total volume count and free volume count
under the data directory line, so a volume size limit that outstrips the
disk is visible at startup instead of surfacing later as failed writes.
* terraform: add cloud-agnostic core renderer module
Renders per-node weed argv, systemd units, config files, disk-mount and secret-fetch scripts, and cloud-init from an address map. Creates zero cloud resources. Flags verified against the weed binary: volume uses -mserver for the master list, gRPC is -port.grpc (auto http+10000), minFreeSpacePercent is a string, filer store via -defaultStoreDir.
* terraform: add mTLS and JWT security module
Generates the CA, per-component certs with distinct CNs, and JWT signing keys via the tls/random providers. Emits a core_security object plus PEMs for secret-store delivery.
* terraform: add AWS deployment module and examples
Reserves stable ENIs first, renders config via the core, then creates instances, prevent_destroy EBS data disks mounted at /data, and the cluster security group. With enable_security, generates certs/JWT, stores them in SSM SecureString, grants an instance role, and fetches them at boot so secrets stay out of user_data. Keyed for_each on every stateful tier.
* terraform: add local cluster test harnesses
run_local_cluster.sh and run_local_secure.sh render a cluster with the core and run real weed processes, asserting master quorum, volume registration, filer/s3 round-trips, mutual-TLS formation, and JWT enforcement. Use an isolated high port range with a guard so they never touch a cluster already running on the machine. The weed binary defaults to $(go env GOPATH)/bin/weed.
* terraform: add CI workflow and README
fmt/validate/tofu-test plus smoke jobs that build weed and run both harnesses.
* terraform: guard against empty filesystem UUID in mount script
An empty UUID made grep -q match any fstab line, skipping the fstab entry and breaking the mount. Fail fast when blkid returns no UUID.
* terraform: sanitize cluster name in WEED_CLUSTER env keys
Hyphens or spaces in cluster_name produced invalid systemd/bash env var names; map non-alphanumerics to underscores.
* terraform: omit empty jwt.signing block from security.toml
With enable_security and no JWT key, the template emitted [jwt.signing] key="". Gate the block on a non-empty key and cover it with a test.
* terraform: mark core security input as sensitive
The security object carries JWT signing keys; keep them out of plan output and known values.
* terraform: enforce jwt_length minimum of 32
* terraform: note region/AZ coupling in HA example
* terraform: guard WORKDIR before recursive delete in test harnesses
* terraform: fix README fence language and test count
* terraform: handle embedded s3 with no filer nodes
Indexing sort(keys(var.filers))[0] errored at plan time when embedded S3 was enabled but no filers were defined; fall back to an empty config source.
* terraform: scope kms:Decrypt to a configurable key arn
Replace the hardcoded Resource="*" with a kms_key_arn variable (default "*") so production can restrict decrypt to a specific CMK.
* terraform: encrypt EBS data volumes at rest
Set encrypted = true on the volume/filer data disks and the all-in-one example disk.
* terraform: protect filer instances from API termination
Filers hold the leveldb2 metadata store, so they are stateful and get the same disable_api_termination as masters and volumes.
* terraform: stop instance before detaching in all-in-one example
* terraform: drop stale references to the removed plan doc
* terraform: correct stale mount-step comment in aws module
* terraform: mark Terraform support as experimental in README
GET/HEAD object with an explicit versionId that does not exist returned
NoSuchKey. AWS S3 returns NoSuchVersion (404) for this case; tools that
distinguish "key gone" from "this version gone" rely on that code.
Add the ErrNoSuchVersion error code and use it on the GET and HEAD
specific-version lookups. Only a genuine not-found maps to NoSuchVersion;
a transient or internal filer error now maps to InternalError (500)
instead of a misleading 404. getSpecificObjectVersion wraps its lookup
error with %w so callers can detect filer_pb.ErrNotFound.
Add a per-row Export button (files and folders) that downloads the filer
metadata in the length-prefixed FullEntry protobuf format that weed shell
fs.meta.load reads, gzipped as <name>.meta.gz like fs.meta.save. Folders are
walked recursively via the filer BFS metadata stream, excluding the system
log subtree. Streamed over gRPC so it keeps working with the filer HTTP
listener disabled.
withFilerClientFailover treated a filer's ErrNotFound like a transport
failure: it kept the result, re-queried every other filer, and finally
wrapped the answer as "all filers failed, last error: ... no entry is
found in filer store".
For workloads with many legitimate misses (e.g. GET object?versionId=X
for a version that was deleted or expired), this turned each 404 into N
filer round-trips and produced a misleading error string.
A reachable filer that answers ErrNotFound has given an authoritative
answer; failover exists to route around unreachable or unhealthy filers,
not to look harder for an entry the store reports as absent. Return
ErrNotFound directly instead of fanning out. Callers that need
read-after-write retries already handle that at the S3 semantic layer
(e.g. getLatestObjectVersion).
* perf(s3.iam.GetUser): Make the API default to the request username if not
specified
This makes the Embedded S3 IAM API align with the documented behavior of the AWS IAM
API as per AWS Docs: https://docs.aws.amazon.com/IAM/latest/APIReference/API_GetUser.html
BREAKING CHANGE: This changes the default behavior of the Embedded IAM API to use the
username of the user holding the accesskey used to make the request in
the GetUsername request handler.
* test: cover GetUser implicit username default
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* operation: bound upload retries and honor context cancellation
retriedUploadData hardcoded 3 attempts and an uninterruptible backoff
sleep. A synchronous replica write to a dead host therefore paid the
full dial timeout three times over before failing.
Add UploadOption.MaxAttempts (<=0 keeps the default of 3) so callers can
cap attempts, and make the loop return as soon as the context is
cancelled so an abandoned upload unwinds instead of retrying.
* topology: fail replica writes fast when a replica is unreachable
DistributedOperation already returns on the first error, but a single
dead replica is itself the slow result: its goroutine retries the upload
three times through the dial timeout (~30s) before any error surfaces,
stalling the originating client write the whole time.
Make the replica write a single attempt (MaxAttempts=1) so a dead
replica fails after one dial timeout instead of three, and thread a
context into DistributedOperation that is cancelled once the outcome is
decided, so a healthy replica is no longer held hostage by one stalled
in a dial. The originating client write is what retries.
* topology: keep replica deletes off the client request context
ReplicatedDelete runs after the local needle is already deleted. Driving
the replica deletes off r.Context() means a client disconnect cancels
them and orphans needles on the replicas, so use a background context.
* operation, topology: trim comments on the replica fail-fast path
* fix: return immediately on first error in DistributedOperation
* simplify DistributedOperation fail-fast to a single buffered channel
Drop the separate errCh: the collector now fails fast on the first error
it reads off the buffered resultCh and returns ret.Error(), so the early
return carries the same [host]: err annotation as the aggregated path and
there is no select race between two channels.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
A volume removed from writables by the write-assign path stamps fullSince
in RecordAssign, which UpdateVolumeSize's recovery branch needs to re-add
it once it decays back under the limit. A volume removed by the heartbeat
capacity path (SetVolumeCapacityFull) never stamped it, so after the
reported size dropped — vacuum, TTL expiry, bulk deletes — the volume
stayed out of writables forever, even though every heartbeat carried the
smaller size.
Stamp fullSince when the capacity path actually removes a volume from
writables, so the existing recovery branch fires. Gating on the removal
keeps it paired with the caller's activeVolumeCount decrement, matching
RecordAssign. Oversized volumes still stay out, as before.
* fix(vacuum): notify master writable after worker vacuum commit
Add Phase 3 (markWritableOne) that walks vacuumTargets and calls
VolumeMarkWritable on each replica's volume server, mirroring
batchVacuumVolumeCommit's per-replica SetVolumeAvailable. Failures are
logged at WARN; the task does not fail because the vacuum itself
already succeeded. See upstream seaweedfs#9685.
* fix(vacuum): delay Phase 3 to let post-commit heartbeats settle
Phase 3's VolumeMarkWritable can race with the volume server's first
post-commit heartbeat. SetVolumeWritable adds the vid to writables,
but a racing heartbeat whose ReadOnly value changed re-runs
EnsureCorrectWritables against the master's per-replica cache, and any
replica still cached as ReadOnly=true silently removes the vid again
— with no further heartbeat change to trigger another recovery.
Sleep 30s after Phase 2 (Commit) so every replica's post-vacuum
heartbeat has reached the master before Phase 3 fires. Cancel cleanly
on ctx.Done so a shutdown during the wait still exits.
* fix(vacuum): reduce post-commit settle from 30s to 10s
VolumePulsePeriod is 5s, so 10s (2x) is enough margin for every
replica's post-commit heartbeat to reach the master before Phase 3
fires. 30s was overly conservative and made TestVacuumExecutionIntegration
hit its 30s context deadline.
* fix(vacuum): use flat 1m timeout for VolumeMarkWritable RPC
VolumeMarkWritable on the volume server is a metadata operation
(reopen idx + flags + master ReadOnly=false heartbeat), independent
of volume size. Scaling via vacuumTimeout(time.Minute) gave it tens
of minutes — even hours on TB volumes — so a single unresponsive
replica could block Phase 3 indefinitely. Use a flat 1m cap.
* fix(vacuum): gate post-vacuum mark-writable on commit read-only state
Phase 3 force-called VolumeMarkWritable on every replica unconditionally,
clearing the read-only flag and persisting ReadOnly=false even for a
replica left read-only by an operator, an EIO quarantine, or low disk.
That overrode states the master deliberately keeps out of writables;
master built-in vacuum gates the same step on the commit's IsReadOnly via
SetVolumeAvailable.
Capture the VacuumVolumeCommit response and skip Phase 3 when any replica
came back read-only, letting it recover on its own ReadOnly=false
heartbeat. Drop the 10s post-commit settle sleep: the heartbeat race it
guarded needed a replica cached read-only at the master, which the gate
now excludes.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* wdclient, dailyrun: add equal jitter to retry backoff
Prevents thundering-herd retries when many clients recover from a
transient failure at the same instant (e.g., filer restart, network
partition healing).
Uses equal jitter: wait in [d/2, d) instead of deterministic d.
This bounds the maximum wait while still desynchronizing clients.
Files:
- weed/wdclient/filer_client.go (LookupVolumeIds retry loop)
- weed/s3api/s3lifecycle/dailyrun/dispatch.go (dispatchWithRetry)
Tests added for bounds, zero/negative inputs, and distribution sanity.
Closes#9735
* wdclient: honor ctx cancellation during LookupVolumeIds backoff
---------
Co-authored-by: Mohamed Chorfa <mohamed.chorfa@thalesgroup.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Adds standard Kubernetes liveness/readiness endpoints to all HTTP
servers that were missing them:
- S3: adds /readyz (already had /healthz)
- IAM: adds /healthz and /readyz (had none)
- Volume: adds /readyz (already had /healthz)
- Filer: adds /readyz on default and readonly mux
- Master: adds /healthz and /readyz at root level
(preserves existing /cluster/healthz)
All endpoints reuse existing health handlers or return 200 OK as a
minimal foundation. Future PRs can enhance /readyz with dependency
checks without breaking the contract.
Closes#9736
Co-authored-by: Mohamed Chorfa <mohamed.chorfa@thalesgroup.com>
On a read-only watched path the idle heartbeat keeps sync_offset fresh,
but a busy source filer still emits a MaxUnsyncedEvents marker after many
filtered events. The marker has a non-nil but empty EventNotification, so
the client routed it to the event path, where it advanced no real
watermark yet drove offsetFunc to republish the stale processed
watermark — regressing the gauge between heartbeats and spiking the
derived lag every time a filtered-event burst landed.
Route the empty marker through OnIdleHeartbeat like the idle heartbeat so
its fresh timestamp keeps the gauge current; it still advances the
in-stream resume cursor.
* fix(shell): verify volume.merge output before overwriting replicas
volume.merge overwrote every replica with the merged copy without checking it was complete. Read back the merged copy and refuse to overwrite unless it holds at least as many live needles as the most complete source replica, leaving the originals intact on a short or empty merge.
* fix(shell): keep merged volume until all replicas are rebuilt
On a copy failure partway through the overwrite loop, the temporary merged copy was deleted along with the half-rebuilt replicas. Stop deleting it until every replica has been rebuilt; on failure the verified copy is kept so the merge can be re-run to completion.
* refactor(shell): reuse readVolumeStatus in ensureVolumeReadonly
* fix(shell): guard against nil volume status response
A bearer-token client whose SDK appends a CRC32 trailer sends an
unsigned-streaming PUT (STREAMING-UNSIGNED-PAYLOAD-TRAILER) with no SigV4
signature, so getRequestAuthType classifies it as authTypeStreamingUnsigned.
The auth dispatch ignored the bearer token and fell back to anonymous, and
newChunkedReader tried to verify the bearer token as a SigV4 seed signature
and failed, so the body could not be decoded either.
Dispatch the streaming-unsigned auth on whatever credential is present
(SigV4 / JWT / anonymous), and skip the SigV4 seed-signature recompute for
JWT requests in the chunked reader.
* fix(volume): verify the .dat-tail needle in the integrity check
CheckVolumeDataIntegrity checked the last entry by file position in the .idx
and, for a live needle, flipped the volume read-only when fileSize > fileTailOffset.
That entry is the .dat tail only when the .idx is in append order; a key-sorted
.idx (weed fix and other rebuilds listed entries by key) puts the highest-key
needle last, whose tail sits mid-file, so healthy volumes went read-only on every
load and re-running weed fix only reproduced the sorted index.
Locate the needle at the maximum offset — the one physically last in the .dat —
and verify the .dat ends exactly at it, regardless of .idx ordering. The
append-ordered common case stays O(1) (the last entry's on-disk end matches the
.dat size); only a key-sorted index pays a single linear scan. Deletion
tombstones at the tail are now verified too, instead of skipping the file-size
check.
* fix(command): weed fix rebuilds the .idx in .dat offset order
SaveToIdx wrote entries via AscendingVisit — sorted by key, the .sdx/.ecx shape
— so the rebuilt .idx put the highest-key needle last instead of the .dat-tail
needle, and dropped tombstones whose live needle was gone. Collect the live and
deleted entries, sort by .dat offset, and write them in append order so the .idx
stays a faithful log whose last entry is the real .dat tail.
Modern botocore attaches a CRC32 trailer to plain PutObject, turning the
payload into STREAMING-UNSIGNED-PAYLOAD-TRAILER. An anonymous upload then
carries that header but no Authorization, so it was classified as
authTypeStreamingUnsigned and sent straight to SigV4 verification, which
rejected it as AccessDenied while explicit credentials kept working.
Fall back to the anonymous identity when an unsigned-streaming request
carries no signature, mirroring the plain anonymous path. The request
stays classified as unsigned-streaming so the chunked body is still
decoded.
* test(framework): support multiple disks per server in MultiVolumeCluster
StartMultiVolumeClusterWithDisks gives each volume server N data
directories (one DiskLocation each), passed to -dir as a comma list, with
a per-server disk-dir accessor for file inspection. StartMultiVolumeCluster
keeps its one-disk default.
* test(ec): end-to-end encode over a multi-server multi-disk stuck layout
A volume in the stuck state — real .dat source, a 0-byte stub replica, and
partial stale EC shards from an interrupted encode — must converge to one
valid EC layout. Asserts the full shard set across servers, .ecx/.vif kept
per server (info file survives the source-volume delete), stale shards
cleared, and no regular .dat/.idx left behind.
* fix(storage): keep EC .vif when deleting a coexisting regular volume
A regular volume and an EC volume for the same id share <base>.vif. When
EC shards are distributed onto a server that still holds the regular
volume — the encode source, or any replica the planner targets — the
post-encode VolumeDelete ran removeVolumeFiles and stripped the shared
.vif, leaving the freshly built EC volume without its info file.
Skip the .vif in removeVolumeFiles when an EC volume for the same id
exists on the disk (mounted, or a sealed .ecx on disk). The regular
volume's .dat/.idx still go; the EC sidecars survive.
A two-server end-to-end test encodes a volume whose source and a stub
replica both also receive shards, and asserts the final on-disk layout:
both .dat/.idx gone, each server holding only its assigned shards plus
.ecx/.vif. Storage unit tests cover the with-EC and no-EC cases, and the
Rust seaweed-volume port carries the same guard and tests.
* test(storage): assert .idx is removed in the no-EC destroy case
Strengthen TestDestroyRemovesVifWhenNoEc to confirm the full regular
volume cleanup (.dat, .idx, .vif) when no EC volume coexists.
* fix(filer): derive inodes by hash instead of a snowflake sequencer
Compute the same inode the FUSE mount would: non-hard-linked entries hash path + crtime, hard links hash their shared HardLinkId so every link resolves to one inode. Removes the snowflake inodeSequencer and the SEAWEEDFS_FILER_SNOWFLAKE_ID knob; inodes are now deterministic across filers.
* chore: remove the experimental NFS gateway
The NFS frontend ('weed nfs') was the only consumer of the inode->path index. Remove the weed/server/nfs package, the command and its registration, the integration test harness, and the CI workflow; go mod tidy drops the willscott/go-nfs and go-nfs-client dependencies.
* refactor(filer): drop the inode->path index
With the NFS gateway gone, nothing reads it. A regular file's inode is a pure hash of its path and a hard link's is a hash of its shared HardLinkId -- both derivable on demand -- so the secondary KV index and its write/remove hooks are dead. Removes filer_inode_index.go and the recordInodeIndex hooks from the store wrapper.
* add make test_keylock_s3 for local develop and debug
* fix typos
* add condition oidc:azp
* docker: reuse test/s3/iam realm and iam config for keycloak dev compose
Point the keycloak dev compose at the existing test/s3/iam configs instead
of a parallel realm/port/key/role set. Adds one declarative realm import
(seaweedfs-test-realm.json) as the single realm source and drops the
duplicated iam.json/s3.json.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(ec): delete empty stub replicas before distributing EC shards
An interrupted encode can leave a 0-byte .dat replica behind. Until now
the only thing that removed it was deleteOriginalVolume, which runs after
distribute+mount and calls VolumeDelete -> removeVolumeFiles. A regular
volume and an EC volume share the same <collection>_<vid>.vif path, so
deleting the stub at that point strips the .vif out from under the
freshly distributed shards.
Sweep the original replicas with VolumeDelete(OnlyEmpty=true) before
distribute: doIsEmpty uses the same superblock threshold, so only the
0-byte stubs go and any data-bearing replica is refused and kept for the
post-verify delete. Servers cleared in the sweep are skipped by
deleteOriginalVolume so it never touches a server that now holds only EC
shards.
* fix(ec): fail the encode when an empty-replica sweep can't confirm a node
The sweep swallowed every VolumeDelete(OnlyEmpty) error, so a transient
failure on a stub node fell through to the post-verify force-delete on
that node — the shared-.vif clobber the sweep exists to avoid.
Treat only the expected cases (volume not empty, or already gone) as
leave-in-place; any other error propagates and fails the encode, which
rolls back the readonly marks and retries next cycle.
* fix(ec): prefer credible replica as canonical metric in EC detection
An interrupted encode can leave a 0-byte .dat replica behind. When that
stub sits on a lower-sorting server than the real replica, the
lowest-server canonical pick reported Size=0, tripped the min-size gate,
and the volume was stranded in skippedTooSmall: detection never proposed
an encode, so the partial EC shards were never cleared and re-distribute
kept hitting the mounted-volume guard.
selectCanonicalMetric now prefers the lowest-server credible replica
(data-bearing, not already EC), falling back to the lowest-server metric
only when nothing is credible so the downstream gates skip as before. A
leftover EC shard set on a lower server no longer short-circuits the
volume at the IsECVolume guard either, so the orphan-source cleanup and
re-encode paths get their chance.
* fix(ec): treat a bare superblock .dat as a stub too
An interrupted encode or copy can write the 8-byte superblock and then
fail, leaving an 8-byte .dat with no data. isStubReplica used a strict <
so that file slipped through as credible, could win the canonical pick on
a low server, and re-tripped the min-size gate. Use <= the superblock so a
data-less .dat never shadows a real replica.
* fix(s3): honor MetadataDirective=REPLACE for system metadata on CopyObject
* fix(s3): match copy metadata keys case-insensitively for legacy data
Legacy / non-S3 write paths (FUSE mount, direct filer HTTP API, older
versions) may persist Cache-Control etc. in lowercase form. Make
isManagedCopyMetadataKey case-insensitive so mergeCopyMetadata still
clears stale source values under REPLACE, and let the COPY branch of
processMetadataBytes fall back to a lowercase key on the source so
legacy values survive into the destination (re-emitted as canonical).
Mirrors the existing x-amz-meta-* backward-compat path.
* fix(s3): keep legacy non-canonical tag and system metadata across COPY
The previous case-insensitive isManagedCopyMetadataKey caused
mergeCopyMetadata to delete legacy lowercase x-amz-tagging-* and
mixed-case system headers, but the COPY branch in processMetadataBytes
only matched canonical or strict-lowercase keys when re-populating
them, so any non-canonical key was permanently dropped on COPY.
- COPY now scans existing in a single pass and uses strings.EqualFold
against the system header whitelist, re-emitting under the canonical
header name. Handles any case folding (CACHE-CONTROL, Cache-control,
etc.), not just strings.ToLower.
- COPY tagging branch now uses hasPrefixFold(k, AmzObjectTagging) and
re-emits the canonical X-Amz-Tagging-<suffix>, mirroring the existing
X-Amz-Meta-* migration path.
- Tests cover lowercase/uppercase/mixed-case system headers and tags
surviving COPY.
* fix(s3): make COPY of system metadata and tags deterministic across case variants
Single-pass EqualFold matching let Go's randomized map iteration pick
either the canonical or a legacy-cased value when both lived on the
source, so the COPY result varied between calls.
Both COPY branches now use two passes: a canonical-exact lookup first,
then a case-insensitive fallback that only writes when the canonical
slot is still empty. Mirrors the collision-check pattern used by the
X-Amz-Meta-* migration path.
Tests run the canonical-vs-legacy collision 32 times each to exercise
varied map orders.
* fix(s3): apply REPLACE Content-Type on in-place copy
The metadata-only self-copy path never set Attributes.Mime, so a same-key
CopyObject with REPLACE and a new Content-Type silently kept the old type.
Route in place only when the Mime is unchanged; otherwise take the locked
clone path (still metadata-only, reuses source chunks) and set the new Mime
there. Also covers the versioned self-copy path.
* perf(s3): drop per-key ToLower in isManagedCopyMetadataKey
Use the allocation-free hasPrefixFold helper instead of lowercasing the key
and both constant prefixes on every metadata-key check.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* chore(weed/storage/backend/s3_backend): remove unused function
* fix(s3_backend): cache session under the composite region|endpoint key
createSession looked up sessions by region|endpoint but stored them by
region alone, so the cache never hit and a new session was built every
call. With getSession gone the lock can also drop to a plain Mutex.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(shell): don't halt volume.fsck purge on a stuck read-only volume
A failed VolumeMarkWritable on one volume aborted the entire fsck purge
run; per-volume errors now log and continue so remaining volumes still
get purged.
* fix(shell): unify volume.fsck per-volume skip logging at the caller
Return the mark-writable error from purgeOneVolume instead of logging
in two places — the caller already prints "skip purging volume N: %v"
and defers still fire on the error return.
* fix(shell): collect volume.fsck purge-skipped volumes and report at end
Track volume IDs whose purge was skipped (mark-writable failure or
other per-volume errors) and print a sorted summary so operators don't
have to scrape the run log to find them. Deletes for those volumes are
already skipped; this just makes them explicit.
* fix(s3tables/iceberg): make metadata spec-compliant and accept real-world manifest names
Two related issues prevent SeaweedFS S3 Tables from interoperating with
strict Iceberg clients (Java/Spark/Flink/Trino):
1. iceberg-go v0.5.0 serializes empty TableMetadata state by dropping
keys via `omitempty` on optional pointer/slice fields. The Iceberg
table spec, however, requires `current-snapshot-id`, `snapshots`,
`snapshot-log`, `metadata-log`, and `refs` to be present even when
empty (`current-snapshot-id` must be -1 for a table with no
snapshots). Java's TableMetadataParser uses JsonUtil.getLong on
`current-snapshot-id` and throws "Cannot parse missing long
current-snapshot-id" against responses produced by this server.
2. The Iceberg layout validator only accepts manifest filenames that
match Iceberg's internal naming (`{uuid}-m{n}.avro`,
`snap-{n}-{n}-{uuid}.avro`). Real writers — notably Flink's sink —
emit manifests like
`{flink-job-id}-{checkpoint}-{operator-id}-{n}.avro`, which the
validator rejects with 403, breaking INSERT commits.
Fixes:
* Add ensureMetadataSpecCompliance helper that backfills the five
spec-required empty-state fields when iceberg-go omits them or emits
explicit JSON null. Apply it on every code path that writes
v*.metadata.json to S3 or returns metadata to clients
(handlers_table create-table, handlers_commit, commit_helpers
create-on-commit, plus MarshalJSON on LoadTableResult and
CommitTableResponse). Real values from non-empty tables are never
overwritten.
* Add catch-all regex entries to metadataFilePatterns accepting any
*.avro / *.metadata.json filename composed of [A-Za-z0-9._-]. The
Iceberg spec does not mandate filename format; the strict patterns
remain for documentation. Metadata-directory subdirectory rejection
and the data-file path validation are unchanged.
No upstream dependencies are forked: iceberg-go stays at v0.5.0 and
go.mod is untouched. The compliance layer can be removed once upstream
emits spec-compliant output.
Tests (all pass under `go test -race`):
- metadata_compliance_test.go: 5 cases covering missing fields,
preserved real values, explicit null, invalid JSON, empty input.
- iceberg_layout_test.go: 3 groups (16 subtests) covering real-world
manifest names from Flink/Spark/Iceberg, security boundary
(subdirectories, bad extensions), and data-file regression.
* fix(s3tables/iceberg): preserve metadata key order and keep config field stable
Two small follow-ups on the spec-compliance fix:
* ensureMetadataSpecCompliance now splices missing keys in at the byte
level just before the closing brace, so iceberg-go's struct-declared
key order survives the backfill. The previous unmarshal/remarshal
through map[string]json.RawMessage silently alphabetized every key in
the document, which is spec-legal but breaks byte-equality fixtures
and any downstream hashing of the persisted metadata. The slower
remarshal path is kept for the rare explicit-null replacement case.
* LoadTableResult.MarshalJSON now serializes Config without omitempty,
matching the struct field tag. The custom marshaler had silently
flipped the tag to ,omitempty, which made the "config" key disappear
from the response whenever s3Endpoint was unset (since
buildFileIOConfig returned an empty but non-nil Properties map).
Tests:
- PreservesOriginalKeyOrder pins the byte-level output against
iceberg-go's emitted shape; would have caught the alphabetization
regression.
- EmptyObjectBackfilled covers the {} -> sentinels-only case (no
leading comma).
- AllPresentReturnsSameBytes confirms the no-op path returns input
bytes unchanged, with whitespace intact.
- iceberg_layout_test pins the catch-all $ anchor: metadata/file.avro.txt
must still be rejected.
* fix(s3tables/iceberg): guard ensureMetadataSpecCompliance against top-level null
json.Unmarshal of a JSON `null` literal succeeds but leaves the map nil.
The current byte-append path no-ops gracefully on this input, but the
slow remarshal path would panic with "assignment to entry in nil map"
if the input ever combined `null` with the explicit-null detection. Add
an explicit nil-map short-circuit so the safety property is obvious
from the source, and a test that pins the contract.
* test(s3tables/iceberg): assert full byte equality in AllPresentReturnsSameBytes
The prefix check only caught a missing "{\n " opener, so the test
would have passed even if the function silently reordered keys or
collapsed whitespace later in the document. Switch to a full string
comparison so any future regression in the no-op path is loud.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(filer/postgres): default to ON CONFLICT upsert to keep tx alive
A KvPut from the inode-index secondary write could fail with 23505
(duplicate key) inside a rename's transaction, after which the next
statement returned 25P02 and rename surfaced to FUSE as EIO. Default
the postgres upsert query when enableUpsert=true so INSERTs are
idempotent; the enableUpsert=false escape hatch is preserved for
non-PG-compatible backends.
* fix(filer/mysql): default to ON DUPLICATE KEY UPDATE upsert
Same shape as the postgres default: when enableUpsert=true but no
upsertQuery is configured, install a sensible default so the
inode-index KvPut does not waste a duplicate-key roundtrip on every
entry write. Uses the VALUES() form so the default works on MariaDB
and MySQL >=5.7; the MySQL 8.0.19 row-alias form is left to explicit
config.
* fix(filer): default enableUpsert=true for sql stores
The default-template fallback only kicks in when enableUpsert=true,
so minimal configs that omit the flag entirely were still exposed.
Default it on for postgres/postgres2/mysql/mysql2; an explicit false
in filer.toml still wins because SetDefault only fills absent keys.
fix(remote_storage/gcs): forward entry.Attributes.Mime as ContentType
Same gap as the S3 client: filer.remote.sync to GCS never populated
the object's ContentType, so HTML/CSS/etc. ended up stored as the GCS
default and didn't render correctly in browsers.
Mirrors the existing Azure client behavior
(weed/remote_storage/azure/azure_storage_client.go).
* fix(replication/s3sink): forward entry.Attributes.Mime as ContentType
Same gap as the remote_storage S3 client: filer.replicate uploads via
s3manager.Uploader without populating ContentType, so replicated objects
on S3-compatible backends (e.g. Backblaze B2) store binary/octet-stream
and browsers refuse to render HTML, CSS, etc.
Pass entry.Attributes.Mime through to UploadInput.ContentType, leaving
the header unset when no Mime is recorded so the remote keeps its own
default.
* fix(replication/s3sink): nil-guard entry.Attributes when reading Mime
* Revert "fix(replication/s3sink): nil-guard entry.Attributes when reading Mime"
This reverts commit 08c3698e44.
The function already dereferences entry.Attributes.Mtime and
entry.Attributes.Md5 unconditionally on the same path, so a nil guard
on Mime alone is inconsistent and provides no real safety.
fix(remote_storage/s3): forward entry.Attributes.Mime as ContentType
filer.remote.sync was uploading every object without a Content-Type, so
S3-compatible backends (e.g. Backblaze B2) stored binary/octet-stream
and browsers refused to render HTML, CSS, etc.
Pass entry.Attributes.Mime through to UploadInput.ContentType, leaving
the header unset when no Mime is recorded so the remote keeps its own
default behavior.
LastIndex returns -1 when the separator is missing and can return any
position when both are present. A path like /vol/file.jpg,abc gives
dotSep<commaSep, so path[commaSep+1:dotSep] slices with start>end and
panics. Only treat the dot as an extension boundary when it sits after
the comma.
* fix(util): guard BytesToUint{16,32,64} against short input
length is computed as uint, so length-1 on an empty slice underflows
to MaxUint and the loop indexes b[0] on a zero-length slice. BytesToUint16
also indexed b[0]/b[1] with no length check. All call sites today gate
the slice length explicitly, so this hardens the API for new callers
rather than fixing a live crash.
Return 0 on short input, matching the existing variable-length contract.
* BytesToUint16: match variable-length contract of the 32/64 helpers
A 1-byte slice should return uint16(b[0]) rather than 0, matching how
BytesToUint32 and BytesToUint64 treat short input.
* fix(vacuum): batch all replicas in a single plugin worker task
The plugin worker vacuum path emitted one TaskDetectionResult per
(volume, server) replica, but the dispatcher gates duplicate tasks per
volume via ActiveTopology.HasAnyTask. The first replica's task was
created and the remaining N-1 replicas were silently dropped, so only
one replica per volume was ever vacuumed — leaving the others with all
their garbage intact.
Mirror the master built-in flow (topology.vacuumOneVolumeId →
batchVacuumVolumeCheck/Compact/Commit/Cleanup) by:
- aggregating detection metrics by VolumeID so a single task carries
every replica in TaskParams.Sources
- having VacuumTask accept []string servers (instead of a single
string), re-check each replica's garbage ratio at execute time to
derive a vacuumTargets subset, and run Compact/Commit/Cleanup against
only that subset
- updating the dispatcher (plugin_handler.Execute, register.CreateTask)
to forward every Sources node to NewVacuumTask
* fix(vacuum): run all-replica vacuum in two phases to keep failure atomic
The prior implementation iterated Compact → Commit → Cleanup against
each replica in sequence. A Compact failure on the second replica left
the first one already committed (its active files swapped with the
.cp* files), producing replica divergence with no automatic recovery.
Split performVacuum into two phases, matching topology.vacuumOneVolumeId:
Phase 1 — Compact all targets. If any fails, run VacuumVolumeCleanup
on every target to drop the .cpd/.cpx/.cpldb temp files, then abort.
No replica has swapped yet, so every replica returns to its original
state.
Phase 2 — Commit all targets. Best-effort, matching
batchVacuumVolumeCommit: per-replica errors are collected and
surfaced together. Once any replica has swapped there is no clean
rollback, so a partial Phase 2 failure requires operator
reconciliation.
Adds compactOne / commitOne / cleanupOne / cleanupAll helpers and
removes the old performVacuumOne.
* fix(vacuum): abort when any replica's garbage check fails
The prior check tolerated per-replica RPC errors and only failed the
task if every replica errored — partial failures were silently treated
as "ineligible" so the responding replicas would still be vacuumed.
That produces divergence the moment the unreachable replica comes
back: it still carries the original garbage while the others have
been compacted.
Match topology.batchVacuumVolumeCheck's contract instead — its return
value (errCount == 0 && len(vacuumLocationList.list) > 0) gates the
whole vacuum on every replica's check succeeding. If any replica is
unreachable or its VacuumVolumeCheck RPC errors, abort the task; the
volume will be retried on the next detection cycle once the replica
is healthy.
* fix(vacuum): guard against nil metrics and TaskSource entries
Detection's bucket-building loop dereferenced m.VolumeID without
checking m for nil. VacuumTask.Validate built sourceSet from
params.Sources without checking each entry for nil. Both paths would
panic on a malformed protobuf payload that managed to deliver a nil
slot. Skip nil entries in both loops — neutral with the existing
nil/empty filtering already done in register.CreateTask and
plugin_handler.Execute.
* test(vacuum): success path no longer calls VacuumVolumeCleanup
The plugin worker vacuum is now two-phase (Compact-all → Commit-all,
with Cleanup only invoked on Compact failure to roll back .cp* temp
files). This matches topology.vacuumOneVolumeId, where
batchVacuumVolumeCleanup runs only on the Compact-failure branch.
On a successful Commit the temp files do not linger:
- CommitCompactVolume renames .cpd → .dat and .cpx → .idx
- leveldb needle map renames .cpldb → .ldb (needle_map_leveldb.go)
so calling VacuumVolumeCleanup afterwards is a redundant no-op. The
prior worker code called it unconditionally and the integration test
asserted that — switch the expectation to cleanupCalls == 0 to
document the new (and master-aligned) contract.
* fix(filer.sync): resolve manifest chunks against source filer
`UpdateEntry` was passing `filer.LookupFn(fs)` — the sink filer client —
into `compareChunks`. But `oldEntry`/`newEntry` chunks come from the
source cluster, so manifest resolution must hit the source filer's
volume servers. With two clusters that have overlapping volume IDs
(common once they grow past a few hundred volumes), the sink lookup
returns its own volume's URLs and the fetch 404s on the source's
fileKey:
compare chunks error: fail to read manifest 631,0babe...: 404 Not Found
The 404 aborts the diff, the manifest chunk never gets replicated, and
the target ends up with whatever flat chunks happened to land from
earlier partial syncs — visible as `SIZE_MISMATCH` in filer.sync.verify
on files large enough to use chunk manifests (~150 GB+ in practice).
Only the manifest path was wrong; flat-chunk reads in `fetchAndWrite`
already use `fs.filerSource.ReadPart`.
* trim comment
* test(filer.sync): regression test for source-filer manifest lookup
Two recording filer gRPC servers stand in for source and sink. Driving
UpdateEntry with a manifest chunk and observing which one receives
LookupVolume proves compareChunks routes source-side lookups through
fs.filerSource, not fs. Reverting the fix flips the call onto the sink
filer and fails the assertion.
* drop test
* fix(filer.sync): validate chunk size in FilerSink to prevent 0-byte propagation
FilerSink.fetchAndWrite previously trusted the source response and the
upload result blindly: a 200 OK / Content-Length: 0 reply from a broken
source volume was happily uploaded as a 0-byte needle to the destination,
and the destination filer metadata was then written with the source
chunk size. The result was permanent silent corruption -- ls shows the
file at its original size but reads fail with EIO.
Add two cheap defenses inside fetchAndWrite:
1. After assembling fullData, compare its length against sourceChunk.Size.
2. After a successful upload, compare uploadResult.Size against
sourceChunk.Size.
Both checks wrap a new sentinel errChunkSizeMismatch that the retry
callback recognizes and refuses to retry -- needle.size=0 on disk is a
persistent state, not a transient network error, so the sync should stop
loudly on the affected entry instead of looping or, worse, silently
propagating it.
Tests:
* TestValidateReplicatedChunkSize -- table-driven coverage of healthy,
legitimately empty, zero-byte read, short read, and truncated upload
cases.
* TestFetchAndWriteRejectsZeroByteSource -- end-to-end: an httptest
source that returns 200 OK with an empty body must cause fetchAndWrite
to return errChunkSizeMismatch after exactly one source hit (fail
fast, no retry storm).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* filer.sync: bubble size-mismatch past CreateEntry/UpdateEntry
Three follow-ups on the chunk-size validation:
- Use %w in replicateOneChunk so the errChunkSizeMismatch sentinel
survives the wrap and reaches errors.Is callers up the stack.
- In FilerSink.CreateEntry/UpdateEntry, surface errChunkSizeMismatch
instead of warning-and-nil. Other errors (deleted source chunk,
transient network) keep the existing swallow so a hiccup doesn't
stall the stream.
- Drop validateReplicatedUploadSize: uploadResult.Size is set
client-side from the same len(fullData) we already validated
pre-upload, so the second check can't fail.
Test: scope the RetryWaitTime override to the one test that needs it,
add a regression that locks in the errors.Is chain through
replicateChunks.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* wdclient: prune filers dropped from master discovery
Filer discovery only appended new addresses; it never removed ones that
disappeared from the master snapshot. After a K8s filer pod rolled to a
new IP the old address lingered in filerAddresses and got retried again
every resetTimeout window, stalling S3 uploads on i/o timeouts.
Treat the master snapshot as authoritative: keep survivors (preserving
their health counters and the active round-robin index), append newcomers
with fresh health, drop the rest. Empty snapshots are still ignored so a
transient master outage can't wipe the list.
* wdclient: skip discovery snapshots with no usable addresses
Guard against the defensive case where master returns updates whose
addresses are all empty; reconciling against an empty discovered set
would prune every filer.
* fix(volume): avoid nil-deref when needle map loader errors
A corrupt .idx whose size is not a multiple of NeedleMapEntrySize sends
the read-only load path into NewSortedFileNeedleMap, which returns
(*SortedFileNeedleMap)(nil) when reverseWalkIndexFile rejects the file.
The multi-value assignment `v.nm, err = NewSortedFileNeedleMap(...)`
parks that typed-nil pointer in the v.nm NeedleMapper interface, so the
subsequent `v.nm != nil` guard still passes — and the post-load
MaxNeedleEnd structural check dispatches through the promoted mapMetric
accessor on a nil receiver, segfaulting the whole volume server at
load time.
Reset v.nm explicitly after every loader failure so the interface is
truly nil, and skip the MaxNeedleEnd check when err is non-nil since
the value would come from a partial walk anyway. NewLevelDbNeedleMap
has the same typed-nil-on-error shape and is fixed the same way.
* fix(volume): close indexFile when needle map load errors
Pre-fix the typed-nil v.nm path either leaked indexFile silently
(SortedFileNeedleMap.Close had a nil-receiver early return) or crashed
(LevelDbNeedleMap.Close had no such guard). With v.nm cleared to nil
on error, the defer cleanup no longer calls Close at all, so the
LoadCompactNeedleMap success-with-error path now also leaks indexFile.
Close indexFile explicitly on each loader error to keep ownership
balanced.
* trim comments
Random allocation could pick 33646 = admin.port (23646) + GrpcPortOffset.
weed mini reserves that as Admin's gRPC port even when the test only
overrides Master/Filer/S3/Iceberg, so the explicit Filer flag failed
with "reserved for gRPC calculation" and TestRisingWaveIcebergCatalog
flaked. Pre-seed the reserved set with every mini default HTTP port
plus its +10000 offset so a random pick (or its own gRPC offset) cannot
land on a service the caller left at its default.
* master: timeout AllocateVolume/DeleteVolume and defer growRequest cleanup
The volume-grow goroutine clears the layout's growRequest flag only after
ms.DoAutomaticVolumeGrow returns, and AllocateVolume / DeleteVolume were
calling the volume-server RPC with context.Background(). A volume server
that hung mid-call (heavy I/O, stuck lock, dead peer behind a stable VIP)
would park the goroutine forever, leaving growRequest=true and silently
blocking every subsequent automatic grow for that layout — Assign retries
then drained their 30s budget with "context deadline exceeded" until the
operator restarted the master.
Bound both RPCs with a 5-minute deadline (creating/removing a volume is
sub-second normally, generous for contended disks) and move the flag
clear + filter delete into defers so a panic in DoAutomaticVolumeGrow
doesn't strand the layout either.
* allocate_volume: shorten timeout to 1m for faster recovery
Volume create/delete is sub-second under normal conditions; 1 minute is
generous even on a contended disk and clears the growRequest flag well
before too many client Assigns drain their own retry budget.
* trim comments
KvPut/KvGet/KvDelete bypassed store.getKey(), so filer.store.id and
other KV writes landed outside the configured prefix. With a Redis
ACL restricted to the prefix this errored with NOPERM; without the
ACL the keys silently lived in the wrong namespace.
* writeJson: drop unused JSONP branch
No in-tree caller uses ?callback=. Always serve application/json
with X-Content-Type-Options: nosniff.
* seaweed-volume: drop unused JSONP branch
Mirror Go: always serve application/json with
X-Content-Type-Options: nosniff.
* writeJson: drop unreachable StatusNotModified check
bodyAllowedForStatus already returns early for 304.
* test/volume_server: rename and rewrite JSONP test to assert callback is ignored
CI: /status?callback=myFunc now returns plain application/json
with X-Content-Type-Options: nosniff.
* s3,iceberg: reject `..`/NUL in URL path vars
Both gateway routers use mux.NewRouter().SkipClean(true), so a request like
`GET /bucket-A/../evil-bucket/key` survives routing as bucket=bucket-A,
object=../evil-bucket/key. The captured key is then joined into a filer path;
util.JoinPath / path.Join collapse the `..` server-side and the read lands in
evil-bucket. With auth on, IAM still authorizes against bucket-A (the mux var),
so policy is evaluated against the wrong target.
Add a middleware on the S3 bucket subrouter and the Iceberg REST router that
rejects any `.`, `..`, NUL, or — for single-segment slots — embedded slash in
the captured path vars before any handler runs. NormalizeObjectKey already
folds `\` to `/` and decoding happens in mux, so `%2e%2e` and `..\` are caught.
* s3,iceberg: reject empty captured vars and empty namespace parts
Comma-ok the var lookup so we only check captured slots, then treat an empty
captured value as a rejection on its own — downstream path.Join would
otherwise collapse it and let the next segment pick the bucket.
For iceberg, also reject empty parts after splitting the namespace on \x1F so
leading/trailing/consecutive unit separators (which parseNamespace silently
folds out) don't let distinct route values collapse to the same parsed
namespace.
Register loggingMiddleware before validateRequestPath on the iceberg router
so rejected requests still produce an audit-log line.
Adds two FUSE integration tests on the existing dlm cluster harness (the
-dlm mounts route advisory locks to the owner filer):
- TestPosixLockCrossMount: an flock taken on one mount blocks the other,
and is grantable after release — the routed-to-owner path end to end.
- TestPosixLockSurvivesFilerLoss: hold flocks on many files, stop filer1
so keys it owned migrate to filer0; after the ring settles and the
holding mount re-asserts, every lock is still honored. Asserts only the
settled state; the transient migration window is unit-covered.
Locks are taken on read-only fds so the -dlm whole-file write lock (a
different mechanism, held until close) isn't involved. Skipped on
non-Linux: only Linux forwards advisory locks (SETLK) to the FUSE server;
macFUSE handles flock in-kernel per mount.
The codec (Set.Marshal/Unmarshal) and its posix_lock.proto were built to
let the lock set ride in an inode's entry metadata, but the authority is
in-memory and ownership handoff/restart is handled by mounts re-asserting
their held locks over the RPC — neither serializes the set. Nothing calls
the serde outside its own tests, so drop it (codec, proto, generated pb,
Makefile). The in-memory Set/Manager are unchanged.
After a (re)start the owner defers would-be grants for posixLockWarmup
while mounts re-assert, trusting only locally-visible conflicts, so it
does not double-grant from empty state; a deferred grant is a retry for
SetLkw and EAGAIN for non-blocking SetLk, never a spurious grant. Cooling
now fail-closes: if the previous owner is unreachable during a ring
change, defer rather than risk a double-grant. readyAt is atomic so the
handler reads it without locking.
While the ring changed within the last snapshot interval, a fresh owner
asks the key's previous owner (LockRing.PriorOwner) whether it still
holds a conflicting lock before granting TRY_LOCK or answering GET_LK, so
it does not double-grant before re-assertion rebuilds its local state.
The probe is marked cooling_probe so the previous owner answers from
local state without recursing. PriorOwner uses the snapshot's prebuilt
ring rather than rebuilding a hash ring per call.
* mount: renew POSIX lock leases via keepalive
The mount tracks the inode keys it holds locks on and a background loop
renews its session lease (KEEP_ALIVE) with each key's owner filer every
5s, within the filer's 15s TTL. A live mount is never reaped; a dead one
stops renewing and owners reclaim its locks. Tracking is a superset:
holds are added on grant and dropped only on owner release, so a still
held lock is never under-renewed.
* mount,filer: re-assert held POSIX locks via keepalive
The owner filer holds POSIX advisory locks as in-memory soft state, so a key's
owner change (ring rebalance) or an owner restart lost or stranded them: the new
or restarted owner was blind to existing holders and would double-grant.
Make the keepalive carry the mount's held lock ranges per key. The mount mirrors
its own granted locks (posixOwn), and each tick re-asserts them to the key's
current owner, which rebuilds that session's locks from the assertion — self
-healing after a takeover or restart. The owner arbitrates re-asserted locks
against other sessions so it never double-grants; a lock that lost a migration
race is reported, not forced. A bare keepalive (no ranges) still just renews.
* filer: session lease + reaping for POSIX locks
A mount renews its session lease by keepalive (new KEEP_ALIVE op); the
owner filer records last-seen per session and a background sweeper reaps
the locks of leased sessions that stop renewing — a dead or partitioned
mount. Only sessions that have renewed are leased, so this is inert until
mounts run with -posixLock.
* mount: route POSIX advisory locks to the owner filer (-posixLock) (#9665)
mount: route POSIX advisory locks to the owner filer under -dlm
With -dlm, GetLk/SetLk/SetLkw and the flush/release cleanup paths go to
the inode's owner filer via the PosixLock RPC instead of the local table,
so flock/fcntl are honored across mounts. Advisory locking rides the same
switch as whole-file write coordination — and is therefore off under
writeback cache, which implies single-writer. The mount calls its filer
and relies on filer-side forwarding to reach the owner. Keys are the inode
identity (HardLinkId else path); SetLkw is client-side polling with the
FUSE cancel channel (no server wait queue); a per-mount session id
namespaces owners; a local hint avoids a release RPC on every close.
* mount,filer: bound posix-lock release RPCs and stop the reaper on shutdown
The unlock/release RPCs run off the syscall path (close/flush) and used
context.Background() with no deadline, so a slow or unreachable filer could
hang close() indefinitely; bound them to 5s (they still aren't cancelled by
an interrupt). The lease-reaping sweeper now selects on a stop channel that
FilerServer.Shutdown closes, instead of looping for the process lifetime.
routedReleasePosixOwner dropped the local owner hint before sending
RELEASE_POSIX_OWNER, so a transient RPC failure left the lock held on the
owner filer with no local record to retry from — stranded until session-lease
reaping. Drop the hint only after a successful release; on failure keep it so
a later flush retries, with lease reaping as the backstop.
With -dlm, GetLk/SetLk/SetLkw and the flush/release cleanup paths go to
the inode's owner filer via the PosixLock RPC instead of the local table,
so flock/fcntl are honored across mounts. Advisory locking rides the same
switch as whole-file write coordination — and is therefore off under
writeback cache, which implies single-writer. Keys are the inode identity
(HardLinkId else path); SetLkw is client-side polling with the FUSE cancel
channel (no server wait queue); a per-mount session id namespaces owners;
a local hint avoids a release RPC on every close. Background unlock/release
RPCs are bounded so a stuck filer can't hang close().
* mount: hold the entry lock while reading chunk size in GetAttr/SetAttr
Async upload workers append chunks to an open handle's shared entry under
the LockedEntry lock (FileHandle.AddChunks), but GetAttr and SetAttr
computed FileSize by iterating entry.Chunks without taking it. A concurrent
append that reallocated the backing array tore the slice read and crashed in
filer.TotalSize. Surfaces with -writebackCache, where handles stay open and
flush asynchronously while metadata ops keep arriving.
Take the LockedEntry lock for those reads (and SetAttr's truncate rewrite).
* mount: re-read entry under the lock in GetAttr/SetAttr
If SetEntry swapped the handle's entry pointer between maybeReadEntry and the
lock acquisition, the old pointer is orphaned. Re-read fh.entry.Entry under
the lock so SetAttr mutates the live entry instead of losing the update, and
GetAttr reports the current one.
* mount: cover the truncate path in TestAttrChunkRace
Alternate SetAttr between mtime-only and a shrinking size so the test also
exercises the entry.Chunks rewrite under fh.entry.Lock, not just the read-side
size walk.
* mount: snapshot chunks under the entry lock on the read path
readFromChunks holds fh.entryLock (excludes SetAttr) but not the LockedEntry
lock the async uploader appends under, so IsInRemoteOnly, the FileSize
fallback, and the RDMA/peer chunk walks read entry.Chunks while AddChunks
reallocated it — the same torn-slice crash as GetAttr/SetAttr.
Snapshot size, inline content, and the chunk list under a brief LockedEntry
RLock, then hand the snapshot to the RDMA/peer helpers instead of holding the
lock across network I/O. The captured slice stays valid: append never mutates
the old backing array, and truncate is excluded by the fh.entryLock.
* filer: in-memory POSIX lock authority (Manager)
Concurrent multi-inode authority over the per-inode Set: a Set per opaque
inode key (path, or hl:<HardLinkId>) plus a session->keys index so a dead
mount's locks reap in O(locks held). Lock state stays in memory like the
distributed lock manager's, off the replicated meta-log. TryLock/Unlock/
GetLk/ReleasePosixOwner/ReleaseFlockOwner/ReleaseSession; empty sets and
stale index entries are pruned on release.
* filer: routed PosixLock RPC over the in-memory authority
Adds the PosixLock RPC (try/unlock/get_lk + the flush/release owner
drops) that the owner filer answers from its in-memory Manager. The
request key is the inode identity ring key; a non-owner filer forwards
one hop (is_moved-bounded), mirroring ObjectTransaction, so the owner's
table stays the single authority under a stale ring view. Strictly
non-blocking; SetLkw polling lives in the mount.
Concurrent multi-inode authority over the per-inode Set: a Set per opaque
inode key (path, or hl:<HardLinkId>) plus a session->keys index so a dead
mount's locks reap in O(locks held). Lock state stays in memory like the
distributed lock manager's, off the replicated meta-log. TryLock/Unlock/
GetLk/ReleasePosixOwner/ReleaseFlockOwner/ReleaseSession; empty sets and
stale index entries are pruned on release.
* filer: POSIX advisory lock set primitive (phase 1)
Pure per-inode conflict/coalesce/range-split logic for fcntl byte-range
and flock whole-file locks, extracted from the mount's PosixLockTable
without its wait queue or inode-map concurrency. Owner identity is
(Sid, Owner) so the same FUSE owner on different mounts never aliases,
and ReleaseSession reaps a dead mount's locks. The owner filer will hold
one Set per inode under the per-path lock; no concurrency control here.
* test: tolerate transient FUSE invisibility in ConcurrentReadWrite
A concurrent truncating overwrite leaves a short-lived dentry/cache window
where the file is momentarily ENOENT to another opener. Retry the reads and
writes a few times before failing, as ConcurrentDirectoryOperations does.
* filer: serialize the POSIX lock set for entry metadata
Versioned fixed-width binary encoding of a Set, so an inode's held locks
can ride in its entry metadata: a lock op materializes the Set from the
blob, applies under the per-path lock, and writes it back. Empty set
encodes to nil so a lock-free inode carries no blob.
* filer: encode the POSIX lock set as protobuf
Replace the hand-rolled fixed-width codec with a LockSetProto message, so the
metadata blob can gain fields without a format-version migration. proto.Unmarshal
already rejects a malformed blob, so the explicit version and length checks go
away. Marshal now returns an error to match.
* filer: POSIX advisory lock set primitive (phase 1)
Pure per-inode conflict/coalesce/range-split logic for fcntl byte-range
and flock whole-file locks, extracted from the mount's PosixLockTable
without its wait queue or inode-map concurrency. Owner identity is
(Sid, Owner) so the same FUSE owner on different mounts never aliases,
and ReleaseSession reaps a dead mount's locks. The owner filer will hold
one Set per inode under the per-path lock; no concurrency control here.
* test: tolerate transient FUSE invisibility in ConcurrentReadWrite
A concurrent truncating overwrite leaves a short-lived dentry/cache window
where the file is momentarily ENOENT to another opener. Retry the reads and
writes a few times before failing, as ConcurrentDirectoryOperations does.
A disconnect/reconnect race could drop a volume from vid2location while it stayed in the data node's disk map, so it showed in volume.list and the admin UI but LookupVolume returned "volume id not found" and never self-healed (the full heartbeat only registered volumes new to the disk map). The full heartbeat now re-registers any reported volume missing from the lookup index, reusing the already-resolved VolumeLayout.
A non-owner filer forwards the whole transaction to the ring owner of route_key, so the owner's per-path lock stays the single serialization point even when the caller's ring view is stale. is_moved bounds forwarding to one hop. The gateway stamps route_key on every routed builder via the shared objectRouteKey helper. Completes taking S3 object mutations off the distributed lock.
* admin: add -metricsPort flag to expose Prometheus metrics
The admin command had no metrics endpoint, so passing -metricsPort
(as the operator does for spec.admin.metricsPort) crashed the process
with "flag provided but not defined". Wire up -metricsPort/-metricsIp
and start the shared Prometheus metrics server, matching filer, master,
and volume.
* admin: emit maintenance task and worker fleet metrics
Add Prometheus metrics for the admin server's distinctive work: the
maintenance task queue and the worker fleet that executes it.
Task lifecycle: maintenance_tasks_by_status / _by_type gauges (snapshot
of the queue), maintenance_tasks_completed_total{type,outcome} counter
and maintenance_task_duration_seconds{type} histogram (recorded when a
task reaches a terminal state), and last/next scan timestamp gauges.
Worker fleet: workers_connected and worker_slots{used,max} gauges, plus
worker_events_total{event} counting register/unregister/stale removals.
Gauges are snapshotted by a background goroutine on the admin server;
counters and the histogram are recorded at their event sites.
* admin: read worker slot totals under lock, clear next-scan gauge when idle
GetWorkers returns live worker pointers; summing CurrentLoad/MaxConcurrent
outside the queue lock races with task assignment and completion. Add
GetWorkerSlotTotals to aggregate under the lock.
Also reset maintenance_next_scan_timestamp_seconds to 0 when the scanner
is not running, so it can't retain a stale value after a stop.
Probe one throwaway write once per process before the lifecycle tests run, absorbing the post-start volume-growth window so the first real PutObject doesn't race volume growth and 500. Each call is bounded by the remaining 60s budget; CreateBucket is retried within it.
A non-versioned metadata-only self-copy (CopyObject with source == destination
and the REPLACE directive) is a read-modify-write of one entry, which is why it
held the distributed lock. It now routes to the owner as a serialized
PATCH_EXTENDED: the owner merges the new managed metadata (set the replacements,
delete the dropped keys) onto a fresh read of the entry under its per-path lock,
so a concurrent change to non-managed keys (legal hold, retention, version id) is
preserved instead of clobbered, and bumps mtime.
PATCH_EXTENDED gains touch_mtime for the mtime bump. Versioned and suspended
self-copies create a new version (already routed via the copy finalize) and the
no-owner bootstrap keep the lock.
A version-specific DELETE (real version or the null version, including
object-lock WORM-checked ones and governance-bypass) now runs as one routed
transaction on the object's owner instead of holding the distributed lock.
For a real version: recompute the .versions pointer excluding the version
(repoint-before-delete, so a crash leaves a recoverable orphan rather than a
dangling pointer), then delete the version file, under the object's per-path lock.
The null version is the regular object entry, deleted directly (no pointer).
Object-lock buckets gate the delete on the version's WORM guards evaluated on the
owner: legal hold (always) + retention (while not elapsed). Governance bypass
scopes the retention guard to COMPLIANCE mode, so the filer allows a
governance-mode delete while still denying compliance and legal hold — the
gateway never reads the version.
Three primitives make this expressible:
- ObjectTransaction.condition_key: evaluate the condition against a named entry
(the version) while the lock stays on lock_key (the object).
- Recompute.exclude_name: omit a child from the scan, to repoint before delete.
- WriteCondition.Clause gate_key/gate_value: scope IF_EXTENDED_TIME_ELAPSED to a
mode, expressing governance bypass without a gateway-side read.
completeMultipartUpload routes its writes to the object's owner filer when an
owner is known, off the distributed lock. Idempotent replay is handled
gateway-side in prepareMultipartCompletionState (it returns the existing result
when the object already carries this UploadId), so the lock is not needed to
dedupe retries; with no owner yet, the lock remains as the bootstrap path.
Versioned completion flips the .versions pointer via routedVersionedFinalize
(RECOMPUTE_LATEST). Non-versioned and suspended completion write the object via
routedMkFile (a routed PUT) so the write serializes with concurrent writes to
the same key on the owner's per-path lock. The version file itself is a unique
path and stays a plain mkFile.
s3: route versioned/suspended delete markers and versioned COPY off the lock
createDeleteMarker flips the .versions pointer via routedVersionedFinalize
(RECOMPUTE_LATEST on the owner filer) when an owner is known, so an Enabled or
Suspended DeleteObject takes its pointer flip off the distributed lock; the
delete marker file is written first and the owner re-derives the pointer.
DeleteObjectHandler routes a versioned/suspended delete with no specific version
straight to the owner, off the lock. A specific-version delete and object-lock
buckets keep the lock (the former needs a recompute-after-delete handled
separately; the latter needs gateway-side enforcement).
CopyObject into a versioned bucket finalizes the new version through the same
routed pointer flip.
routableWriteOwner no longer excludes object-lock buckets, so a versioned PUT
(which creates a new version, never overwriting a locked one) and a
non-versioned overwrite (WORM-checked gateway-side before dispatch) route to the
owner filer like any other write.
routedObjectOwner still excludes object-lock: an unversioned object-lock delete
enforces WORM under the lock, so it stays there rather than routing past the
check. Version-specific deletes likewise stay on the lock — routing them needs
the WORM check (on the version entry) and the latest-pointer recompute (on the
object) under one transaction, which the current single condition target cannot
express.
s3: route versioned PutObject finalize off the distributed lock
A versioned write's finalize (flip the .versions pointer to the newest version,
demote the prior latest) now runs as a single RECOMPUTE_LATEST ObjectTransaction
on the object's owner filer, under its per-path lock, instead of the unserialized
updateLatestVersionInDirectory. The version file is written first; the owner
re-derives the pointer by scanning the directory.
RECOMPUTE_LATEST gains size_to_key / mtime_to_key to cache the chosen version's
size and mtime on the pointer, and demote_key / demote_value to stamp the
displaced prior latest (NoncurrentSinceNs for lifecycle) when the pointer moves.
Falls back to updateLatestVersionInDirectory when no owner is known yet.
PutBucketVersioning and PutBucketEncryption ran concurrently each did a
whole-entry read-modify-write of the bucket entry, so one could overwrite the
other's field with a stale copy. Each config write is now a field-level
PATCH_EXTENDED (extended attributes) or set_content (the metadata blob)
ObjectTransaction, routed to the bucket's owner filer and merged onto a fresh
read under its per-path lock. Disjoint fields no longer clobber each other.
s3: route non-versioned object PUT and DELETE off the distributed lock
A non-versioned, non-object-lock object write now goes straight to the key's
owner filer as a single-mutation ObjectTransaction, which serializes it with the
owner's per-path lock and evaluates the precondition, instead of taking a
cluster-wide lock. PUT and DELETE use the object's full path as the lock key, so
a concurrent create and delete of the same key serialize against each other.
The fast path is taken only when the precondition reduces to clauses the filer
can evaluate (existence and a single strong-ETag match); time-based conditions,
ETag lists, weak ETags, post-create hooks, and an unknown owner fall back to the
lock. A routed mutation error other than a failed precondition also falls back,
so the lock path stays the authority for the cases it alone covers.
PrimaryForKey returns "" until the ring view arrives, keeping writes on the lock
until routing is known.
* s3: dial the object lock's primary filer directly
The S3 object write lock builds a fresh short-lived lock per write, each
starting at the seed filer. When the seed isn't the key's hash-ring primary
the filer forwards the request to the primary, and in multi-cluster setups
that forward crosses clusters on every write.
Give the lock client a view of the filer lock ring, fed by the master's
LockRingUpdate broadcasts the gateway already receives, so it dials the
primary directly. The view tracks filer membership by version; a stale view
stays correct because the filer still forwards as a fallback.
Also send the initial ring snapshot to S3 clients, not just filers.
* s3: subscribe to lock-ring updates before starting the master loop
The master delivers the initial LockRingUpdate once, on connect. Registering the
callback after KeepConnectedToMaster started left a window where that first
update could arrive before the handler was set and be dropped, delaying the ring
view until the next membership change. Build the lock client and register the
callback in the masters block before launching the loop; the filers block reuses
that client (or creates a plain one when no masters are configured).
* lock_manager: build the hash ring in a deterministic server order
rebuildRing ranged over the server set (a map), whose iteration order is
randomized per process. On a vnode hash collision the last writer into
vnodeToServer wins, so two nodes holding the same server set could resolve the
collision to different servers and disagree on the primary for keys near that
slot. Now that the S3 gateway also computes PrimaryForKey, such a disagreement
would route the same key to different filers and defeat per-path serialization.
Iterate the servers in sorted order so the ring is identical on every node with
the same set, regardless of discovery order.
* lock_manager: skip redundant ring rebuilds, trim comments
SetRing now ignores a non-zero version at or below the current one once a ring
exists, so repeated LockRingUpdate broadcasts on reconnect no longer rebuild the
ring.
* s3: hold the lock-ring client on the server for route-by-key
Store the object-write lock client on S3ApiServer so handlers can resolve a
key's owner filer via PrimaryForKey.
2026-05-24 00:40:43 -07:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* filer: let PATCH_EXTENDED replace Entry.content
PATCH_EXTENDED merges extended attributes under the per-path lock, reading the
entry fresh, so concurrent patches to different keys don't clobber each other.
Some single-key state lives in Entry.content rather than an extended attribute
(e.g. the S3 bucket metadata blob). Add set_content/content to the mutation so a
patch can replace content the same way -- read fresh, set content, preserve the
rest -- letting a content write and an extended-attribute write on the same
entry serialize on the lock instead of racing whole-entry rewrites.
* Update weed/server/filer_grpc_server.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* filer: test set_content FileSize sync; note chosen content-patch approach
Cover the FileSize behavior of a set_content patch: a file's size follows the
new content length (including when it shrinks), a directory's stays zero. Also
document, in the bucket-config design, that extending PATCH_EXTENDED with
set_content is the implemented path for content-backed config.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
CreateEntry starts with a FindEntry to load the current entry. A conditional
CreateEntry already fetched that entry to evaluate the precondition under the
per-path lock, so the create repeated the lookup.
Add an existing *Entry parameter: when non-nil it is used as the current entry
and the internal lookup is skipped; nil keeps the lookup. The gRPC CreateEntry
handler passes the entry it fetched for the precondition, removing the redundant
read while the lock is held. All other callers pass nil.
A multi-object delete spans many keys that route to different owner filers. The
gateway groups keys by owner and sends one batch per owner; the filer applies
each transaction under its own per-path lock, independent of the others.
A failed transaction (precondition or mutation error) is reported in its own
response without aborting the rest, matching S3 multi-object semantics where
each key succeeds or fails on its own. There is no cross-key atomicity, which S3
batch delete does not require.
* s3: use oidc: prefix for trust-policy conditions in IAM example
Trust-policy conditions for AssumeRoleWithWebIdentity see OIDC claims
under the oidc: prefix, so the docker example's bare "roles" key never
matched and denied every web-identity assume against those roles. Switch
the three roles to oidc:roles.
Also document the available trust-policy condition keys (oidc:iss/sub/aud,
oidc:<claim>, aws:FederatedProvider, aws:userid, sts:DurationSeconds) and
note that roleMapping selects the role for direct OIDC bearer auth while
STS uses the explicit RoleArn plus trust policy.
* s3: clarify aws:userid differs between trust policy and request auth
aws:userid is the raw sub claim during trust-policy evaluation, but a
stable sub+iss hash (ComputeParentUser) during S3 request authorization
after the role is assumed. Note both so the two contexts aren't conflated.
Routing object-lock buckets off the distributed lock needs the retention and
legal-hold check to run atomically with the write, under the per-path lock. Move
just the comparison into the filer, not the S3 semantics: two generic clause
kinds on an extended attribute.
IF_EXTENDED_NOT_EQUAL blocks while extended[ext_key] equals ext_value (a legal
hold). IF_EXTENDED_TIME_ELAPSED blocks while extended[ext_key], read as a unix-
second deadline, is in the future against the filer's clock (retention); a
malformed deadline fails safe. The caller composes these from the object-lock
state and, for a governance bypass, simply omits the retention clause once the
bypass is authorized -- the filer makes no authorization decision and keeps no
S3 knowledge.
Deleting a specific version that happens to be the latest needs the new latest
re-derived from the remaining versions, and that scan must run under the same
lock as the delete. The gateway can't do it atomically across RPCs.
Add a RECOMPUTE_LATEST mutation: it scans a directory under the transaction
lock, picks the child that sorts last (descending) or first by name, copies the
mapped extended keys from it into a pointer entry, and stores its name under
name_to_key. An empty directory clears the pointer keys. The filer stays
mechanical and S3-agnostic: the caller, which knows the versioning scheme,
supplies the sort direction and the key mappings. A missing pointer entry is a
no-op, so a replayed transaction is idempotent.
A versioned object write touches several entries that must change together: the
main object, a delete marker or version file, and the latest pointer on the
.versions directory. Holding a distributed lock across separate RPCs to do this
is what the per-path lock was meant to replace, but a single CreateEntry only
covers one entry.
Add ObjectTransaction: a request carries a lock_key (the object path), an
optional WriteCondition, and an ordered list of mutations (PUT / DELETE /
PATCH_EXTENDED). The filer holds the per-path lock on lock_key for the whole
call, checks the condition against the entry at lock_key, then applies the
mutations in order. Callers route the object's writes to its owner filer so the
lock is authoritative across all of the object's entries.
DELETE and PATCH of an absent entry are no-ops, so a replayed transaction is
idempotent. PUT entries are metadata-scoped; data-bearing writes (chunks) are
written before the transaction, as today.
Add an optional WriteCondition to CreateEntryRequest. When set, the filer
evaluates it against the current entry while holding the per-path lock, so the
check and the write are atomic on this filer, and returns PRECONDITION_FAILED
when it does not hold. The caller must route the key's writes to the owner filer
for the check to be authoritative.
A condition is a list of clauses that all must hold (logical AND). One clause is
the common case; several express what a single comparison cannot: an ETag set
(If-Match / If-None-Match with multiple values), weak-ETag comparison, and
compound conditions. ETag comparison mirrors the S3 gateway's precedence (stored
Seaweed ETag attribute, then the Md5/chunk fallback) and follows RFC 7232
strong/weak rules, so results match without coupling the filer to S3 handling.
Condition parsing and evaluation live in filer_grpc_server_condition.go.
CreateEntry is a FindEntry-then-write with no lock, so concurrent creates to the
same path race: OExcl can admit two creators, and a conditional check-then-act
has no atomicity. Add a per-path exclusive lock (util.LockTable, which evicts
idle keys so it stays bounded) on the FilerServer and take it in CreateEntry, so
the existence check and the write are atomic on this filer.
This is the local serialization point that lets callers route a key's writes to
its owner filer and drop the distributed lock for that key. AppendToEntry keeps
its distributed lock for now; it can move to the per-path lock once its callers
route to the owner.
2026-05-23 14:22:42 -07:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* EC detection: build placement snapshot once per cycle, not per volume
planECDestinations rebuilt the full ecbalancer snapshot (FromActiveTopology) for
every eligible volume, and resolved each shard destination's address via
ResolveServerAddress, which rebuilds the whole node map on every call. Both are
O(volumes x topology) and made detection time out on large clusters
(TestErasureCodingDetectionLargeTopology: 300k volumes hit the 2-minute
deadline).
Build the snapshot and the node-address map once per detection cycle and pass
them in. planECDestinations now reserves the shards it assigns directly into the
shared snapshot, so volumes planned later in the same cycle still see the reduced
capacity (previously this was observed by rebuilding from ActiveTopology's
pending tasks). Large-topology detection drops from a 120s timeout to ~3.5s.
* Update weed/worker/tasks/erasure_coding/detection.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Delete the EC placement package and the dead encode planner code
Now that encode (and repair) place via ecbalancer.Place, nothing uses the
erasure_coding/placement package or the EC-only planner machinery
(ecPlacementPlanner, diskInfosToCandidates, calculateECScoreCandidate,
distributeECShards) in detection.go. Removes them and the package, along with the
planner-direct unit tests.
* Add shared super_block.ResolveReplicaPlacement; use it in ec_balance
* Add ecbalancer.FromActiveTopology snapshot constructor for EC encode/repair
* Add ecbalancer.Place greenfield/repair placement core (strict + durability-first)
* topology: add GetEffectiveAvailableEcShardSlots; FromActiveTopology uses shard-granular free slots
GetDisksWithEffectiveCapacity flattens reserved shard slots into volume slots via
integer truncation, so an in-flight EC task reserving a non-multiple-of-
DataShardsCount number of shards was lost from the snapshot and freeSlots was
over-reported. GetEffectiveAvailableEcShardSlots subtracts the full reservation
impact at shard granularity.
* ecbalancer.Place: reject nodes without a free disk of the requested type
FromActiveTopology keeps all disk types in the snapshot, so an SSD-only request
could be routed to a node with only HDD capacity (pickBestDiskOnNode then returns
disk 0 on the wrong tier). Filter rack/node selection to those with a free disk
of the requested type.
* ecbalancer.Place: enforce ReplicaPlacement DiffDataCenterCount (per-DC shard cap)
* ecbalancer: enforce DiffDataCenterCount in balance (cross-DC phase + cross-rack DC cap)
Adds a cross-DC corrective phase that drains data centers holding more than
DiffDataCenterCount shards of a volume, and a per-DC cap on cross-rack move
targets. Both are no-ops when DiffDataCenterCount is unset, so balance output is
unchanged for non-DC placements.
* topology: ratio-aware EC shard slots and provisional empty-disk slot
GetEffectiveAvailableEcShardSlots now takes the target collection's data-shard
count, so a 4+2 volume's larger shards are not over-counted at 10 per volume slot;
and it keeps the one provisional slot for freshly started empty servers that
report max=0, matching getEffectiveAvailableCapacityUnsafe. FromActiveTopology
threads the ratio through.
* ecbalancer.Place: explicit disk-type filter signal (fix HDD vs any ambiguity)
HardDriveType normalizes to "", which collided with "" meaning any disk. Add
Constraints.FilterDiskType and normalize both sides so a hdd request matches disks
reported as "" and never leaks to SSD, while filter=false still means any.
* ecbalancer: add clearShardAccounting for repair snapshot reconciliation
Clears one disk's copy of a shard from per-domain accounting and recomputes the
node-level union (preserving a kept copy on another disk of the same node), without
crediting capacity. Repair uses it to drop to-be-deleted copies before placing
missing shards.
* ecbalancer: don't cap cross-DC target racks when DiffRackCount is unset
len(racks)+1 wrongly limited each target rack (3 in a 2-rack cluster), so draining
a DC could stop short of the DiffDataCenterCount cap. Use MaxShardCount+1 as the
effectively-unlimited default.
* topology/ecbalancer: ratio-correct EC capacity accounting
Reservation shard slots (default ShardsPerVolumeSlot units) are now converted to
the target ratio before subtracting, and existing EC shards are charged by size
(targetDataShards/shardDataShards) so a 2+1 shard isn't counted as one 10+4 slot.
Per-shard ratio lookup is behind shardDataShards (OSS uses the standard ratio).
* ecbalancer.Place: candidate tiering and eligible-rack caps
Adds a per-disk eligibility/preference abstraction so Place supports:
- preferred-tag whole-plan retry (try disks carrying the earliest tags first,
widen to all only if a tier cannot place every shard; reports
SpilledOutsidePreferredTags),
- soft disk-type spill via DiskTypePolicy (Any/Prefer/Require): Prefer fills the
preferred type then spills, reporting SpilledToOtherDiskType; Require filters,
- even per-rack caps that divide by racks holding an eligible disk, so a tiered
cluster (e.g. SSDs in 2 of 4 racks) isn't capped impossibly low.
Disk tags carried via Node.AddDiskTags + FromActiveTopology.
* ecbalancer: export ClearShardAccounting for repair snapshot reconciliation
* ecbalancer: address review feedback (ratio rounding, bitmap walk, same-DC moves)
- topology/ecbalancer: round shard-reservation and existing-shard footprint up
when converting to target-ratio shard slots, so a sub-slot reservation is not
truncated to zero and free capacity is not overstated for low-data-shard
layouts (targetDataShards < ds).
- erasure_coding: add ShardBits.All iterator and use it across the balancer,
cross-DC phase, and placement scoring instead of scanning 0..MaxShardCount and
probing Has on every id.
- ecbalancer: allow same-DC cross-rack moves when a DC already sits at its
DiffDataCenterCount cap; a same-DC move leaves the DC total unchanged. Add a
regression test that fails without the guard.
- ecbalancer cross-DC phase: pick targets via the eligible-aware
pickNodeInRackEligible/pickBestDiskEligible helpers so the disk-type filter is
honored and a 0 disk id is not mistaken for a valid selection.
* ecbalancer: test ecShardSlotsOnDisk fractional round-up
Cover the mixed-ratio path (targetDataShards < existing data shards) so a
shard's fractional footprint is never floored to zero and free capacity is not
overstated. Exercises the round-up via the targetDataShards parameter; OSS uses
the standard ratio at runtime while the enterprise build hits it with real
per-volume ratios.
* ecbalancer: assert node B rack in TestFromActiveTopology
* ecbalancer: split Destination into separate DataCenter and bare Rack
Replace the composite "dc:rack" Rack field on Destination with separate
DataCenter and bare Rack values, matching topology.DiskInfo and the worker-task
convention. Callers (and tests) read the data center directly instead of parsing
the composite with strings.SplitN.
* shell ec.balance: use utilization-based global balancing (parity with worker)
The shell's global rebalance phase balanced by raw shard count; switch it to
fractional fullness (shards/capacity), as the worker already does. On uniform
capacity the two agree; on heterogeneous capacity it fills nodes proportionally
instead of driving small-capacity nodes toward full.
Updates the heterogeneous-capacity regression test to assert even fullness
(~equal shards/capacity per node) rather than even shard count.
* ecbalancer: bounded-proportional per-DC shard spread
DiffDataCenterCount was enforced only as a ceiling (drain-to-cap), which could
leave a within-cap-but-lopsided DC distribution under a loose cap (e.g. 10/4 of 14
with cap=10). Now the cross-DC phase, the cross-rack DC guard, and Place all target
boundedMaxPerDC = min(DiffDataCenterCount, max(ceil(total/numDCs), parityShards)):
shards spread proportionally across DCs, but no tighter than the durability floor
(once each DC holds <= parityShards a DC loss is recoverable, so further spreading
only adds cross-DC/WAN traffic). No-op when DiffDataCenterCount is 0; identical to
before when the cap is the binding constraint.
* ecbalancer: drop DiffDataCenterCount enforcement for EC placement
The 1-byte volume ReplicaPlacement packs xyz into x*100+y*10+z<=255, so the DC
digit can only be 0-2 -- far too small to be a meaningful per-DC EC shard cap (a
cap of 1-2 would demand 7-14 DCs for a 10+4 volume). It's volume replica-placement,
not an EC spec. Removes the cross-DC balance phase, the DC guard in the cross-rack
phase, and the per-DC cap in Place (and the just-added bounded-proportional logic);
EC relies on the RP-independent rack/node even spread instead. Rack/node caps
(DiffRackCount/SameRackCount) are unchanged. Per-domain EC caps are left for a real
EC placement spec.
* ecbalancer: enforce per-disk durability cap; symmetric reserve/release
Place now refuses to put more than parityShards shards of a volume on a single
disk (pickBestDiskEligible skips a disk once it holds parityShards of the volume,
a hard cap not relaxed even in durability-first). Previously Place assigned by
free capacity, so a skewed near-full cluster could pile >parityShards onto one
disk -> losing it loses the volume; only distinct-disk count was checked. This
covers encode and repair (both route through Place); the caller skips/leaves the
volume rather than minting an unrecoverable layout.
Also makes reserveShard decrement freeSlots unconditionally, symmetric with
releaseShard's unconditional increment (the old guarded decrement could credit a
phantom slot on release if a shard were ever reserved onto a full disk).
* ecbalancer: add Topology.ReleaseVolumeShards (clear + credit) for greenfield encode
Releases all of a volume's shards from the snapshot and credits the freed disk
capacity, so a greenfield encode can plan as if stale EC shards from a prior failed
attempt are gone. Safe to credit because the encode task deletes stale shards
(cleanupStaleEcShards) before distributing the new ones. Distinct from
ClearShardAccounting (repair), which does not credit.
* ecbalancer: ReleaseVolumeShards credits node freeSlots, not just disks
releaseShard only increments per-disk freeSlots, but rack capacity is summed from
node freeSlots (buildRacks) and node freeSlots gates node eligibility. Crediting
only disks left a node/rack looking full after releasing stale shards, so a
greenfield encode still couldn't use the freed capacity. Now credits the node by
the total disk-slots freed.
* ecbalancer: correct PlacementMode docs (encode uses durability-first)
PlaceStrict was labeled '(encode)' but encode uses PlaceDurabilityFirst. Clarify
that durability-first is used by both encode and repair, reports relaxations in
PlaceResult.Relaxed, and never relaxes the per-disk durability cap.
* ecbalancer: treat SameRackCount as a direct per-node shard cap
The 3rd ReplicaPlacement digit now caps shards per node at exactly the digit
value, matching how DiffRackCount (2nd digit) caps per rack, instead of allowing
digit+1 per node. This makes the per-rack and per-node caps consistent and
matches the documented "digits cap EC shards per rack and per node" semantics;
e.g. 011 now means at most one shard per rack and one per node.
* EC encode: place shards via ecbalancer.Place + configurable replica placement
Encode now plans destinations through the shared ecbalancer.Place policy
(durability-first: prefers the source disk type and honors replica placement /
caps / anti-affinity, relaxing rather than failing when capacity is tight) instead
of the EC-only placement planner. Targets and capacity reservations use Place's
actual per-disk shard assignment, not a round-robin guess; cross-volume in-cycle
capacity is tracked by ActiveTopology's pending task, so the cached planner is no
longer consulted. Adds a configurable replica_placement (proto field 6 + worker
form + reader) that overrides the master default replication.
The placement-package planner code is left in place (now unused) and removed in a
follow-up that drops the package.
* EC encode: drop unused dataShards param from createECTargets
Addresses review feedback: after switching to Place's per-disk shardsPerPlan
assignment, createECTargets no longer needs the data-shard count.
* EC encode: fix packed-target validation, greenfield stale-shard accounting, RP docs
- Validate counts distinct shard ids across targets, not target rows, so packed
plans (fewer (node,disk) targets than shards) aren't rejected.
- planECDestinations releases the volume's stale EC shards from the snapshot before
Place (ReleaseVolumeShards), crediting their capacity. The encode task deletes
stale shards before distributing, so a retry on tight capacity no longer fails
planning by counting shards that are about to be removed.
- replica_placement config/form help no longer claims a data-center limit (the DC
digit is ignored for EC); detection logs a warning when a DC digit is set.
* EC encode: surface relaxed placement; mark replica_placement best-effort
Encode places with PlaceDurabilityFirst (the chosen lenient behavior), which can
relax caps/anti-affinity/replica-placement to avoid deferring. That was silent
(only disk-type/tag spills were logged). Now logs PlaceResult.Relaxed so a tight
replica placement isn't weakened unnoticed, and the config/form help states the
rack/node caps are best-effort during encode (enforced by rebalancing).
* EC encode: key per-disk shard grouping by struct, not formatted string
planECDestinations grouped destinations using a fmt.Sprintf("%s:%d") map key
per shard; use a {node,diskID} struct key and pre-size the map/slice to the
shard count to drop the per-shard string allocation.
* Add shared super_block.ResolveReplicaPlacement; use it in ec_balance
* Add ecbalancer.FromActiveTopology snapshot constructor for EC encode/repair
* Add ecbalancer.Place greenfield/repair placement core (strict + durability-first)
* topology: add GetEffectiveAvailableEcShardSlots; FromActiveTopology uses shard-granular free slots
GetDisksWithEffectiveCapacity flattens reserved shard slots into volume slots via
integer truncation, so an in-flight EC task reserving a non-multiple-of-
DataShardsCount number of shards was lost from the snapshot and freeSlots was
over-reported. GetEffectiveAvailableEcShardSlots subtracts the full reservation
impact at shard granularity.
* ecbalancer.Place: reject nodes without a free disk of the requested type
FromActiveTopology keeps all disk types in the snapshot, so an SSD-only request
could be routed to a node with only HDD capacity (pickBestDiskOnNode then returns
disk 0 on the wrong tier). Filter rack/node selection to those with a free disk
of the requested type.
* ecbalancer.Place: enforce ReplicaPlacement DiffDataCenterCount (per-DC shard cap)
* ecbalancer: enforce DiffDataCenterCount in balance (cross-DC phase + cross-rack DC cap)
Adds a cross-DC corrective phase that drains data centers holding more than
DiffDataCenterCount shards of a volume, and a per-DC cap on cross-rack move
targets. Both are no-ops when DiffDataCenterCount is unset, so balance output is
unchanged for non-DC placements.
* topology: ratio-aware EC shard slots and provisional empty-disk slot
GetEffectiveAvailableEcShardSlots now takes the target collection's data-shard
count, so a 4+2 volume's larger shards are not over-counted at 10 per volume slot;
and it keeps the one provisional slot for freshly started empty servers that
report max=0, matching getEffectiveAvailableCapacityUnsafe. FromActiveTopology
threads the ratio through.
* ecbalancer.Place: explicit disk-type filter signal (fix HDD vs any ambiguity)
HardDriveType normalizes to "", which collided with "" meaning any disk. Add
Constraints.FilterDiskType and normalize both sides so a hdd request matches disks
reported as "" and never leaks to SSD, while filter=false still means any.
* ecbalancer: add clearShardAccounting for repair snapshot reconciliation
Clears one disk's copy of a shard from per-domain accounting and recomputes the
node-level union (preserving a kept copy on another disk of the same node), without
crediting capacity. Repair uses it to drop to-be-deleted copies before placing
missing shards.
* ecbalancer: don't cap cross-DC target racks when DiffRackCount is unset
len(racks)+1 wrongly limited each target rack (3 in a 2-rack cluster), so draining
a DC could stop short of the DiffDataCenterCount cap. Use MaxShardCount+1 as the
effectively-unlimited default.
* topology/ecbalancer: ratio-correct EC capacity accounting
Reservation shard slots (default ShardsPerVolumeSlot units) are now converted to
the target ratio before subtracting, and existing EC shards are charged by size
(targetDataShards/shardDataShards) so a 2+1 shard isn't counted as one 10+4 slot.
Per-shard ratio lookup is behind shardDataShards (OSS uses the standard ratio).
* ecbalancer.Place: candidate tiering and eligible-rack caps
Adds a per-disk eligibility/preference abstraction so Place supports:
- preferred-tag whole-plan retry (try disks carrying the earliest tags first,
widen to all only if a tier cannot place every shard; reports
SpilledOutsidePreferredTags),
- soft disk-type spill via DiskTypePolicy (Any/Prefer/Require): Prefer fills the
preferred type then spills, reporting SpilledToOtherDiskType; Require filters,
- even per-rack caps that divide by racks holding an eligible disk, so a tiered
cluster (e.g. SSDs in 2 of 4 racks) isn't capped impossibly low.
Disk tags carried via Node.AddDiskTags + FromActiveTopology.
* ecbalancer: export ClearShardAccounting for repair snapshot reconciliation
* ecbalancer: address review feedback (ratio rounding, bitmap walk, same-DC moves)
- topology/ecbalancer: round shard-reservation and existing-shard footprint up
when converting to target-ratio shard slots, so a sub-slot reservation is not
truncated to zero and free capacity is not overstated for low-data-shard
layouts (targetDataShards < ds).
- erasure_coding: add ShardBits.All iterator and use it across the balancer,
cross-DC phase, and placement scoring instead of scanning 0..MaxShardCount and
probing Has on every id.
- ecbalancer: allow same-DC cross-rack moves when a DC already sits at its
DiffDataCenterCount cap; a same-DC move leaves the DC total unchanged. Add a
regression test that fails without the guard.
- ecbalancer cross-DC phase: pick targets via the eligible-aware
pickNodeInRackEligible/pickBestDiskEligible helpers so the disk-type filter is
honored and a 0 disk id is not mistaken for a valid selection.
* ecbalancer: test ecShardSlotsOnDisk fractional round-up
Cover the mixed-ratio path (targetDataShards < existing data shards) so a
shard's fractional footprint is never floored to zero and free capacity is not
overstated. Exercises the round-up via the targetDataShards parameter; OSS uses
the standard ratio at runtime while the enterprise build hits it with real
per-volume ratios.
* ecbalancer: assert node B rack in TestFromActiveTopology
* ecbalancer: split Destination into separate DataCenter and bare Rack
Replace the composite "dc:rack" Rack field on Destination with separate
DataCenter and bare Rack values, matching topology.DiskInfo and the worker-task
convention. Callers (and tests) read the data center directly instead of parsing
the composite with strings.SplitN.
* shell ec.balance: use utilization-based global balancing (parity with worker)
The shell's global rebalance phase balanced by raw shard count; switch it to
fractional fullness (shards/capacity), as the worker already does. On uniform
capacity the two agree; on heterogeneous capacity it fills nodes proportionally
instead of driving small-capacity nodes toward full.
Updates the heterogeneous-capacity regression test to assert even fullness
(~equal shards/capacity per node) rather than even shard count.
* ecbalancer: bounded-proportional per-DC shard spread
DiffDataCenterCount was enforced only as a ceiling (drain-to-cap), which could
leave a within-cap-but-lopsided DC distribution under a loose cap (e.g. 10/4 of 14
with cap=10). Now the cross-DC phase, the cross-rack DC guard, and Place all target
boundedMaxPerDC = min(DiffDataCenterCount, max(ceil(total/numDCs), parityShards)):
shards spread proportionally across DCs, but no tighter than the durability floor
(once each DC holds <= parityShards a DC loss is recoverable, so further spreading
only adds cross-DC/WAN traffic). No-op when DiffDataCenterCount is 0; identical to
before when the cap is the binding constraint.
* ecbalancer: drop DiffDataCenterCount enforcement for EC placement
The 1-byte volume ReplicaPlacement packs xyz into x*100+y*10+z<=255, so the DC
digit can only be 0-2 -- far too small to be a meaningful per-DC EC shard cap (a
cap of 1-2 would demand 7-14 DCs for a 10+4 volume). It's volume replica-placement,
not an EC spec. Removes the cross-DC balance phase, the DC guard in the cross-rack
phase, and the per-DC cap in Place (and the just-added bounded-proportional logic);
EC relies on the RP-independent rack/node even spread instead. Rack/node caps
(DiffRackCount/SameRackCount) are unchanged. Per-domain EC caps are left for a real
EC placement spec.
* ecbalancer: enforce per-disk durability cap; symmetric reserve/release
Place now refuses to put more than parityShards shards of a volume on a single
disk (pickBestDiskEligible skips a disk once it holds parityShards of the volume,
a hard cap not relaxed even in durability-first). Previously Place assigned by
free capacity, so a skewed near-full cluster could pile >parityShards onto one
disk -> losing it loses the volume; only distinct-disk count was checked. This
covers encode and repair (both route through Place); the caller skips/leaves the
volume rather than minting an unrecoverable layout.
Also makes reserveShard decrement freeSlots unconditionally, symmetric with
releaseShard's unconditional increment (the old guarded decrement could credit a
phantom slot on release if a shard were ever reserved onto a full disk).
* ecbalancer: add Topology.ReleaseVolumeShards (clear + credit) for greenfield encode
Releases all of a volume's shards from the snapshot and credits the freed disk
capacity, so a greenfield encode can plan as if stale EC shards from a prior failed
attempt are gone. Safe to credit because the encode task deletes stale shards
(cleanupStaleEcShards) before distributing the new ones. Distinct from
ClearShardAccounting (repair), which does not credit.
* ecbalancer: ReleaseVolumeShards credits node freeSlots, not just disks
releaseShard only increments per-disk freeSlots, but rack capacity is summed from
node freeSlots (buildRacks) and node freeSlots gates node eligibility. Crediting
only disks left a node/rack looking full after releasing stale shards, so a
greenfield encode still couldn't use the freed capacity. Now credits the node by
the total disk-slots freed.
* ecbalancer: correct PlacementMode docs (encode uses durability-first)
PlaceStrict was labeled '(encode)' but encode uses PlaceDurabilityFirst. Clarify
that durability-first is used by both encode and repair, reports relaxations in
PlaceResult.Relaxed, and never relaxes the per-disk durability cap.
* ecbalancer: treat SameRackCount as a direct per-node shard cap
The 3rd ReplicaPlacement digit now caps shards per node at exactly the digit
value, matching how DiffRackCount (2nd digit) caps per rack, instead of allowing
digit+1 per node. This makes the per-rack and per-node caps consistent and
matches the documented "digits cap EC shards per rack and per node" semantics;
e.g. 011 now means at most one shard per rack and one per node.
* fix(s3api/list): cancel ListEntries stream in hasChildren
* fix(s3api): use filer_pb.List in hasChildren
filer_pb.List already wraps the ListEntries stream in a cancellable
context, so the single-entry probe needs no separate helper or manual
context plumbing to avoid the leaked gRPC stream goroutine.
* fix(s3api): propagate request context into hasChildren
Thread r.Context() through listFilerEntries and hasChildren so the
implicit-directory probe cancels when the client disconnects, instead
of running on context.Background().
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
feat(helm): add volume.rust to run the Rust volume server
When set, the volume statefulset execs /usr/bin/weed-volume instead of
'weed volume', dropping the Go-only -logtostderr/-logdir/-v flags and the
'volume' subcommand. All shared flags and extraArgs carry over unchanged.
GitHub Actions artifacts drop the executable bit, so the pre-built Rust
volume server lands in the image as 0644 and 'weed-volume' fails to start
with 'exec: Permission denied'. chmod it 0755 after copying.
* fix(s3): list empty directories as directory markers
A real but empty directory created out of band (mount, mkdir, filer API)
carries no MIME, so it was hidden from S3 listings. hadoop-aws getFileStatus
probes LIST prefix=dir/ &delimiter=/ and reads an empty result as a missing
path, which breaks Spark's eventLog.dir when it points at an empty directory.
Surface such directories as directory markers, matching directories created
via PutObject with a trailing "/". Emptiness comes from the recursion result,
and the marker MIME is set only on the in-memory listing entry, so empty
directories stay eligible for empty-folder cleanup.
* fix(s3): only surface empty directory markers for explicit dir probes
Restrict the empty-directory marker to a trailing-slash prefix probe
(prefix=dir/), the pattern hadoop-aws getFileStatus uses. Plain listings
are left as before, so an empty directory left behind by deleted objects
(e.g. after lifecycle expiration) is no longer shown as a phantom key.
The maintenance scanner tries to plan EC destinations for every
eligible volume, so clusters that can't place EC logged a warning per
volume every cycle. The min-node gate already skips clusters with fewer
nodes than parity shards; demote the rest to V(2).
cluster.check asks every master to ping every volume server, but the
Ping gate validated volume-server targets only against the local
topology. Only the leader receives volume-server heartbeats, so a
follower's topology is empty and every probe through it failed with
"unknown ping target ... of type volumeServer".
Fall back to the volume-server set the master learns over its own
MasterClient subscription to the leader, the same source the filer gate
already trusts. The anti-SSRF intent is preserved: Ping still only dials
recognized cluster members.
fix(storage): harden zero-size idx scrub and mirror to rust
When a zero-size .idx is found, openIndex stats the backing .dat through
v.DataBackend: wrap that GetStat failure with %w, fix the indices typo, and
guard both openIndex and scrubVolumeData against a nil DataBackend (closed or
remote-only volumes) instead of panicking.
Add rust scrub tests for empty (superblock-only .dat, zero-size .idx) and
healthy volumes, keeping the volume server in parity with the go zero-size
scrub handling.
test(volume): wait for master to register both replicas before replicated upload
TestReplicatedUploadSucceedsImmediatelyAfterAllocate allocated the volume on
both nodes via direct AllocateVolume gRPC calls, then uploaded immediately. The
master only learns about replica locations through volume-server heartbeats,
which lag behind those direct gRPC calls, so the replicated write could look up
the master before the second replica was registered and fail with a 500
("replicating operations [1] is less than volume replication copy count [2]").
In production a client obtains its fid from the master assign flow, which
guarantees the master already knows every replica. The test crafts the fid by
hand, bypassing that guarantee, so wait until the master reports both replicas
before uploading.
The disk-spread assertion raced volume growth and heartbeats. volume.grow
-count is a writable-target topup, not add-N, and swallows partial-failure
errors, so one grow could leave a node's data on a single disk; ec.encode
then piles all that node's shards there and ec.balance can't spread them.
Retry grow on under-spread nodes until the master topology shows every node
holding volumes on at least two physical disks, then encode.
* fix(s3): keep server-side copy data in the bucket collection
UploadPartCopy and SSE-C CopyObject assigned destination volumes against
r.URL.Path, the S3 request URI. The filer derives a bucket's collection
only when the assign path sits under its buckets folder, so an S3 URI
routed copied bytes to the default collection instead of the destination
bucket's. Assign against the destination's real filer path.
* refactor(s3): centralize copy-part path and thread dstPath into SSE-C copy
Extract copyPartLocation so the fast path and writeEmptyCopyPart share one
definition of the .uploads/<id>/<n>_copy.part location. Pass the destination
filer path into copyChunksWithSSEC instead of re-deriving it from the request,
and thread it through key rotation so re-encrypt copies also assign in the
destination bucket's collection.
ChunkTransferStatus embeds a sync.RWMutex, so returning a slice of it
made callers copy the lock when ranging. Split out a copyable
ChunkTransferSnapshot holding the data fields and return that instead.
NameList, NameBatch and their serde were an earlier in-memory directory
batch implementation. The redis3 filer store uses its own ItemList
backed by Redis sorted sets, so these types had no production callers
(NameList only via its own test, LoadNameList none at all). Drop them
and the now-orphaned NameBatchData proto message, regenerating
skiplist.pb.go with the repo-standard protoc-gen-go v1.36.6.
* refactor(volume): extract replica sync/select into shared volume_replica package
Move the volume replica reconciliation helpers (status, union builder,
SyncAndSelectBestReplica, ReadNeedleMeta) out of the shell into a new
weed/storage/volume_replica package so both the shell (ec.encode, volume.tier.move,
volume.check.disk) and the EC encode worker can reuse them. No behavior change.
* fix(ec): bring ec.encode worker to parity with the shell
- Sync replicas and encode the most-complete one (via the shared
volume_replica.SyncAndSelectBestReplica) instead of a possibly-stale replica,
marking all replicas readonly first. Prevents silent data loss when a stale
replica is encoded and the originals deleted.
- Skip remote/tiered volumes in detection (shell ec.encode excludes them).
- Min-node safety gate: refuse to encode when cluster nodes < parity shards.
- Align default thresholds with the shell (fullness 0.95, quiet 1h).
* fix(vacuum): plugin path honors min_volume_age_seconds override
deriveVacuumConfig hard-coded MinVolumeAgeSeconds=0, dropping any configured
value. Read it from worker config (default 0, matching the shell/master vacuum
which has no age gate) so an explicit override is honored.
* address review feedback
- config.go: align GetConfigSpec schema defaults (quiet_for_seconds=3600,
fullness_ratio=0.95) with the runtime defaults so UI/bootstrap flows match the
shell (coderabbitai).
- ec_task.go: roll back readonly when markReplicasReadonly fails partway, so
already-marked replicas don't stay readonly (coderabbitai).
- volume_replica: pass the caller's replica statuses into buildUnionReplica instead
of re-fetching them, and skip the per-needle ReadNeedleMeta RPC when the source
replica is read-only (gemini-code-assist).
* test(plugin_workers/ec): make fixtures eligible under the new defaults
The default EC encode thresholds were raised to match the shell (fullness 0.95,
quiet 1h), but the plugin-worker integration fixtures still used 90%-full /
10-minute-old volumes, so detection found no eligible volumes and the tests failed
in CI. Bump the eligible fixtures to 96% full and 2h old.
* fix(redis3): prevent filer crash from inconsistent skiplist ends
DeleteByKey updated the two ends asymmetrically: the start side decided
whether to clear StartLevels[index] by comparing a cached reference key,
while the end side cleared EndLevels[index] structurally. The redis3
ItemList re-keys a node while keeping its id, so that cached key drifts.
When such a node was the only one at a level and got deleted, the stale
key comparison left StartLevels dangling while EndLevels was cleared to
nil. The next InsertByKey then dereferenced a nil EndLevels[0] and took
down the whole filer during a rename or delete.
Match the deleted node by its unique id so both ends stay consistent,
and guard each end in InsertByKey so an already-corrupted skiplist
persisted in Redis self-heals instead of crashing on load.
* fix(redis3): propagate errors from WriteName node split
The case 2.3 split path returned nil instead of the error in seven
branches. Because the split runs a multi-step sequence (DeleteByKey on
the skiplist, ItemAdd, redis range-delete, ItemAdd), a swallowed failure
let WriteName report success while the skiplist was half-updated, which
the caller then persisted - silently corrupting the directory listing
and setting up the very inconsistent-ends state that crashes the filer.
The plugin cluster context forwarded filers as gRPC-only addresses
(host:grpcPort). The admin-script worker stored that in
ShellOptions.FilerAddress, whose shell commands re-derive the gRPC port
via ToGrpcAddress() and re-add the +10000 offset, dialing a non-existent
host:28888.
Carry filers in pb.ServerAddress form (host:httpPort.grpcPort) and let
each consumer convert when it dials: the admin shell uses it verbatim,
while the s3_lifecycle and iceberg workers collapse it to a gRPC address.
Rename the proto field filer_grpc_addresses -> filer_addresses so the
name matches the content.
weed fix -ecx reconstructs the .dat from the local data shards, scans the
needles, and writes a fresh ascending-sorted .ecx containing only live
entries — the same on-disk index WriteSortedFileFromIdx emits at encode
time. When the .vif is also missing it is regenerated from the inferred
EC ratio (flags > .vif > shard-count inference / 10+4) and the .dat size
recovered from the scan.
When some data shards are missing but at least dataShards shards survive,
the missing shards are first reconstructed from the survivors via
Reed-Solomon, so a partial shard set is repaired too.
Also makes erasure_coding.WriteDatFile de-stripe using len(shardFileNames)
instead of the DataShardsCount constant, so the caller's actual data-shard
count is honored (behavior-preserving for the default 10, and fixing the
existing caller that already passes ECContext.DataShards).
This recovers an EC volume whose sealed index was lost from every node
while enough shards survive, a state neither ec.rebuild nor ec.decode can
repair because both require an existing .ecx.
Flags: -ecx, -ecDataShards, -ecParityShards. Run with the volume server
stopped.
* fix(s3): sync IAM policies to advanced IAM Manager policy engine
* test(s3): add unit tests for PutPolicy/DeletePolicy IAM Manager sync
* fix(s3): flush loaded policies in SetIAMIntegration, drop extra reload
Sync the policies already loaded from the credential store into the IAM
Manager's engine from SetIAMIntegration itself, instead of re-running a
full LoadS3ApiConfigurationFromCredentialManager after setup. This covers
both startup orderings without a second filer round-trip or racing the
async loader goroutine: if the load won, the policies are in memory to
push; if SetIAMIntegration won, the load's own sync runs afterward.
Move the runtime PutPolicy/DeletePolicy sync out of the iam.m write lock
so the per-request auth RLock path isn't blocked by the policy recompile.
* fix(s3): serialize IAM manager policy resync to avoid stale snapshots
SyncRuntimePolicies replaces the manager's full policy set, so applying a
policy view captured before a later mutation can resurrect a deleted
policy or drop a new one. Funnel every path (PutPolicy, DeletePolicy,
SetIAMIntegration, and the credential-manager load) through a single
resyncIAMManagerPolicies that serializes on a dedicated mutex and reads
iam.policies fresh at apply time, so the live map always wins regardless
of interleaving. The load now installs the config into iam.policies
before resyncing, closing the window where the manager held policies the
map didn't yet have.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
test(ec): pre-populate disks so multi-disk EC balance spread is deterministic
The multidisk shard-loss regression asserts EC shards spread across more
than one disk per node, but that only holds for disks the balancer can see.
The master enumerates a physical disk only when it already holds a volume
or EC shard — an empty disk leaves no trace, since heartbeats aggregate
capacity per disk type, not per physical disk. So whether the post-encode
balance spread shards depended on how the master happened to place the
filler volumes across disks, which varies by environment: the test passed
locally (shards on 5 disks) but produced one disk per node in CI and failed
the "got 3 disks across 3 nodes" assertion.
Grow a few volumes on each server before encoding so every physical disk
holds a volume and is visible to the balancer. The volume server places
each new volume on its least-loaded disk, so a handful of grows touches
every disk, making the spread deterministic. The assertion still has teeth:
it counts disks holding shard files, so a balancer that failed to spread
would still collapse to one disk per node.
* fix(shell): count physical disks in cluster.status on multi-disk nodes
The master keys DataNodeInfo.DiskInfos by disk type, so several same-type
physical disks on one node collapse into a single DiskInfo entry. cluster.status
(printClusterInfo) and CountTopologyResources counted len(DiskInfos), reporting
one disk per node instead of the real physical disk count, while volume.list and
the admin ActiveTopology already split per physical disk.
Route both counters through DiskInfo.SplitByPhysicalDisk so a node with N
same-type disks reports N. Cosmetic/diagnostic only; placement already uses the
per-disk activeDisk map.
* fix(ec): attribute EC balance source disk per shard and reject same-node moves
On multi-disk nodes the EC balance worker built a node-level view that kept only
the first physical disk id per (node, volume), so a move of a shard living on a
different disk reported the wrong source disk. That source disk drives the
per-disk capacity reservation, so the wrong disk drifts the capacity model the
EC placement planner relies on. Track shards per physical disk and resolve the
actual source disk for every emitted move (dedup, cross-rack, within-rack,
global), keeping the per-disk view consistent as simulated moves are applied.
Also close a data-loss trap: VolumeEcShardsDelete is node-wide (it removes the
shard from every disk on the node) and copyAndMountShard skips the copy when
source and target addresses match, so a same-node move would erase a shard it
never copied. isDedupPhase now requires the same node AND disk, and Validate /
Execute reject same-node cross-disk moves outright.
* fix(ec): spread EC balance moves across destination disks
Port the shell ec.balance pickBestDiskOnNode heuristic to the EC balance
worker so a moved shard is placed on a good physical disk instead of always
deferring to the volume server (target disk 0). The detection now builds a
per-physical-disk view of each node (free slots split from the node total, exact
EC shard count, disk type, discovered from both regular volumes and EC shards)
and, for each cross-rack, within-rack, and global move, chooses the destination
disk by ascending score:
- fewer total EC shards on the disk,
- far fewer shards of the same volume on the disk (spread a volume's shards
across disks for fault tolerance), and
- data/parity anti-affinity (a data shard avoids disks holding the volume's
parity shards and vice versa).
Planned placements are reserved on the in-memory model during a run so multiple
shards moved to the same node spread across its disks rather than piling on one.
* fix(ec): bring EC balance worker to parity with shell ec.balance
The worker's cross-rack and within-rack balancing balanced shards by total
count; the shell balances data and parity shards separately with anti-affinity
and honors replica placement. Port that logic so the automatic balancer makes
the same fault-tolerance-aware decisions as the manual command:
- Cross-rack and within-rack now run a two-pass balance: data shards spread
first, then parity shards spread while avoiding racks/nodes that already hold
the volume's data shards (anti-affinity), mirroring doBalanceEcShardsAcrossRacks
and doBalanceEcShardsWithinOneRack.
- Optional replica placement: a new replica_placement config (e.g. "020")
constrains shards per rack (DiffRackCount) and per node (SameRackCount); empty
keeps the previous even-spread behavior.
- The data/parity boundary is resolved from a per-collection EC ratio (standard
10+4 here), replacing the previously hardcoded constant at the call sites.
Selection is deterministic (sorted keys) to keep behavior reproducible.
* refactor(ec): extract shared ecbalancer package for shell and worker
The EC shard balancing policy was duplicated between the shell ec.balance
command and the admin EC balance worker, and the two had drifted (multi-disk
handling, data/parity anti-affinity, replica placement). Extract the policy into
a new pure package, weed/storage/erasure_coding/ecbalancer, that both callers
share so it cannot drift again.
- ecbalancer.Plan(topology, options) runs the full policy (dedup, cross-rack and
within-rack data/parity two-pass with anti-affinity, global per-rack balance,
and diversity-aware disk selection) over a caller-built Topology snapshot and
returns the shard Moves. It depends only on erasure_coding and super_block.
- The worker builds the Topology from the master topology and turns Moves into
task proposals; the shell builds it from its EcNode model and executes Moves
via the existing move/delete RPCs. Per-collection EC ratio resolution stays in
each caller (passed as Options.Ratio).
- Options expose the two genuine policy differences: GlobalUtilizationBased
(worker balances by fractional fullness; shell by raw count) and
GlobalMaxMovesPerRack (worker moves incrementally across cycles; shell drains
in one pass).
The shell keeps pickBestDiskOnNode for the evacuate command. Policy tests move to
the ecbalancer package; the shell and worker keep their adapter/execution tests.
* fix(ec): restore parallelism and per-type/full-range balancing after ecbalancer refactor
Address regressions and gaps from the ecbalancer extraction:
- Shell ec.balance honors -maxParallelization again: planned moves run phase by
phase (preserving cross-phase dependencies) with bounded concurrency within a
phase. Apply mode does only the RPCs concurrently; dry-run stays sequential and
updates the in-memory model for inspection.
- Rack and node balancing gate on per-type spread (data and parity separately)
instead of combined totals, so a data/parity skew is corrected even when the
per-rack/node totals are even.
- Global rack balancing iterates the full shard-id space (MaxShardCount) so
custom EC ratios with more than the standard total are candidates.
- Cross-rack planning decrements the destination node's free slots per planned
move, so limited-capacity targets are no longer over-planned.
* fix(ec): make EC dedup keeper deterministic and capacity-aware
When a shard is duplicated across nodes, keep the copy on the node with the most
free slots and delete the duplicates from the more-constrained nodes, relieving
capacity pressure where it is tightest. Tie-break on node id so the choice is
deterministic. This unifies the shell and worker (the shell previously kept the
least-free node, an incidental default) on the more sensible behavior.
* fix(ec): restore global volume-diversity and per-volume move serialization
Two more behaviors lost in the ecbalancer refactor:
- Global rack balancing again prefers moving a shard of a volume the destination
does not hold at all before adding another shard of an already-present volume
(two-pass, mirroring the old balanceEcRack), keeping each volume's shards
spread across nodes.
- Shell apply-mode execution serializes a single volume's moves within a phase
while still running different volumes in parallel, so concurrent moves of the
same volume cannot race on its shared .ecx/.ecj/.vif sidecar files.
* fix(ec): key EC balance shards by (collection, volume id)
A numeric volume id can be reused across collections, and EC identity is
(collection, vid) (see store_ec_attach_reservation.go). The ecbalancer keyed
Node.shards by vid alone, so volumes sharing an id across collections merged into
one entry — letting dedup delete a "duplicate" that is actually a different
collection's shard, and letting moves act across collections. Key shards by
(collection, vid) throughout so each volume stays distinct.
* fix(ec): credit freed capacity from dedup before later balance phases
Dedup deletions are simulated only by applyMovesToTopology, which cleared shard
bits but did not return the freed disk/node/rack slots. Later phases reject
destinations with no free slots, so a slot opened by dedup could not be reused in
the same Plan/ec.balance run. applyMovesToTopology now credits the freed
disk/node/rack capacity for dedup moves (non-dedup moves still rely on the inline
accounting their phase already did).
* test(ec): add multi-disk EC balance integration test
Cover issue 9593 end-to-end at the unit level the old tests missed: build the
master's actual multi-disk wire format (same-type disks collapsed into one
DiskInfo, real DiskId only in per-shard records), run it through a real
ActiveTopology and the Detection entry point, then replay the planned moves with
the volume server's true semantics (node-wide VolumeEcShardsDelete) and assert no
EC shard is ever lost. Covers a balanced spread, a one-node-concentrated volume,
and a multi-rack spread, and asserts moves are safe (no same-node cross-disk),
correctly attributed to the source disk, and redistribute concentrated volumes
across both other racks and multiple destination disks.
* fix(ec): aggregate per-disk EC shards when verifying multi-disk volumes
collectEcNodeShardsInfo overwrote its per-server entry for each EcShardInfo of a
volume. A multi-disk node reports one EcShardInfo per physical disk holding shards
of the volume, so only the last disk's shards survived — the node looked like it
was missing shards it actually had. This made ec.encode's pre-delete verification
(and ec.decode) under-count volumes whose shards are spread across disks on one
server, falsely aborting the encode on multi-disk clusters. Union the per-disk
shard sets per server instead.
Also make verifyEcShardsBeforeDelete poll briefly: shard relocations reach the
master via volume-server heartbeats, so a freshly distributed shard set may not be
fully visible the instant the balance returns. Retry before concluding the set is
incomplete; genuine loss still fails after the retries are exhausted.
* test(ec): end-to-end multi-disk EC balance shard-loss regression
Start a real cluster of multi-disk volume servers (3 servers x 4 disks),
EC-encode a volume, run ec.balance, and assert hard invariants the prior
integration tests only logged: after encode all 14 shards exist, ec.balance loses
no shard, shards span more than one disk per node, and cluster.status counts
physical disks (not one per node). This reproduces issue 9593 end to end and would
have caught the multi-disk shard-aggregation bug fixed alongside it.
* fix(ec): bring EC balance worker/plugin path to parity with shell
- Per-volume serialization and phase order: key the plugin proposal dedupe by
(collection, volume) instead of (volume, shard, source), so the scheduler runs
only one of a volume's moves at a time (within a run and against in-flight jobs).
Concurrent same-volume moves raced on the volume's .ecx/.ecj/.vif sidecars; and
because the planner emits a volume's moves in phase order, they now execute in
order across detection cycles, matching the shell.
- disk_type "hdd": normalize via ToDiskType (hdd -> "" HardDriveType) while keeping
a "filter requested" flag, so disk_type=hdd matches the empty-keyed HDD disks
instead of nothing; apply the canonical type to planner options and move params.
- Replica placement: expose shard_replica_placement in the admin config form and
read it into the worker config, mirroring ec.balance -shardReplicaPlacement.
* test(ec): rename worker in-process test (not a real integration test)
The worker-package multi-disk tests build a fake master topology and simulate
move execution; they are not real-cluster integration tests. Rename
integration_test.go -> multidisk_detection_test.go and drop the Integration
prefix so 'integration' refers only to the real-cluster E2Es in test/erasure_coding.
* ci(ec): remove redundant ec-integration workflow
ec-integration.yml duplicated EC Integration Tests under the same workflow name
but ran only 'go test ec_integration_test.go' (one file), so it never ran new
test files (e.g. multidisk_shardloss_test.go) and was a strict, path-filtered
subset of ec-integration-tests.yml, which already runs 'go test -v' over the whole
test/erasure_coding package on every push/PR.
* fix(ec): worker falls back to master default replication for EC balance
For strict parity with the shell, the EC balance worker now uses the master's
configured default replication as the replica-placement fallback when no explicit
shard_replica_placement is set, instead of always defaulting to even spread.
The maintenance scanner reads it via GetMasterConfiguration each cycle and passes
it through ClusterInfo.DefaultReplicaPlacement; detection resolves the constraint
(explicit config wins, else master default, else none) in resolveReplicaPlacement.
A zero-replication default (the common 000 case) still means even spread, so the
common configuration is unchanged.
* fix(ec): plugin path populates master default replication too
The plugin worker built ClusterInfo with only ActiveTopology, so the master
default replication fallback added for the maintenance path never reached
plugin-driven EC balance detection — empty shard_replica_placement still meant
even spread there. Fetch the master default via GetMasterConfiguration (new
pluginworker.FetchDefaultReplicaPlacement) and set ClusterInfo.DefaultReplicaPlacement
so both detection paths resolve replica placement identically to the shell.
* docs(ec): empty shard replica placement uses master default, not even spread
The EC balance config text (admin plugin form, legacy form help text, and
the struct/proto field comments) still said an empty shard_replica_placement
spreads evenly. The runtime resolves empty to the master default replication
(resolveReplicaPlacement), matching shell ec.balance, with even spread only
when that default is empty or zero. Update the text to match and regenerate
worker_pb for the proto comment change.
test(mount): fix fd leak that deadlocked the DLM handoff check
The cross-mount handoff checks held a file open on mount 2 via fd 9 to
keep the distributed lock, then started the SMB writer in a background
subshell. The subshell inherited fd 9, so the SMB writer kept the file
open and waited on a lock held by its own descriptor; the put could
never complete, and the two checks were parked as expected-fail.
Close fd 9 in the subshell (9>&-) so the writer does not hold the file.
The waiter now acquires the freed lock within ~1s, so the two checks are
real assertions and the xfail machinery is gone.
* test(mount): add Samba over FUSE integration test
Export a SeaweedFS FUSE mount over SMB with smbd and drive it with
smbclient: file round-trips, directories, rename, large-file chunking,
recursive upload, cross-protocol consistency, and deletes.
A second -dlm mount adds locking coverage: POSIX fcntl byte-range locks,
distributed-lock write coordination, and concurrent writers. The two
cross-mount handoff checks currently fail and pin a known limitation -
the distributed lock is released on FUSE Release, which the kernel can
delay under contention.
Runs locally via test/samba/run.sh or in Docker via the compose file;
wired into CI as samba-integration.yml.
* fix(cluster): release distributed lock without racing the renewal goroutine
Stop() closed the cancel channel, slept 10ms, then unlocked using
renewToken. A renewal in flight during that window rotates the token on
the server, so the unlock may be sent with a stale token, fail with a
mismatch, and leave the lock to linger until its TTL expires - stalling
other mounts waiting to write the same file.
Wait for the renewal goroutine to exit before unlocking. The channel
close also makes the renewToken read happen-after the last renewal.
* fix(cluster): poll for distributed lock acquisition without exponential backoff
A mount waiting to write a file held by another mount acquired through
util.RetryUntil, whose backoff grows to several seconds. Once the holder
released, the waiter could sleep that long before retrying, stretching
the cross-mount handoff past client timeouts.
Poll at the steady ~1s cadence AttemptToLock already enforces instead.
* test(mount): tighten Samba harness and mark the DLM handoff checks xfail
Run the workflow for weed/cluster changes, fail fast when the filer or
smbd port never opens, and fold the recursive mput result into its own
assertion so it cannot false-pass.
Mark the two cross-mount handoff checks expected-fail: they pin the
remaining DLM liveness bug (the lock is freed only on the delayed FUSE
Release) without failing CI, and turn the suite red if the handoff is
ever fixed.
* fix(cluster): keep a wedged renewal shutdown from sending a stale unlock
If the renewal goroutine is stuck in a slow RPC, Stop() fell through to
unlock anyway once it timed out waiting. A late renewal can rotate
renewToken, so that unlock races it, is rejected on a stale token, and
leaves the lock lingering until its TTL regardless. On the timeout path,
skip the unlock and let the TTL expire the lock instead.
* fix(cluster): wake the long-lived lock renewal loop promptly on Stop
StartLongLivedLock's renewal loop slept uninterruptibly between attempts,
up to 5*renewInterval (2.5*lockTTL) while unlocked. Stop() waits only
lockTTL+2s for the goroutine to exit, so a Stop() during that backoff
would time out before the goroutine woke and closed renewalDone,
breaking the shutdown synchronization. Sleep on a timer with a select on
cancelCh so the loop exits immediately.
* fix(filer): don't disable the SQL idle connection pool when unconfigured
The mysql/mysql2/postgres stores called SetMaxIdleConns(maxIdle)
unconditionally, so an unset connection_max_idle (0) actively kept zero
idle connections - every query opened and closed a fresh connection
instead of reusing the pool.
Only apply the value when it's set; otherwise leave database/sql's
default idle pool of 2 in place.
* comments: shorten idle-pool note
* fix(filer): default the SQL idle pool via config, keep explicit 0 honored
Apply the idle-pool default at the config layer with SetDefault instead of
guarding the SetMaxIdleConns call. An absent connection_max_idle now reads
back as 2 (pool stays on), while an explicit 0 flows through to
SetMaxIdleConns(0) so operators can still disable idle pooling on purpose.
The planner refused to create an EC task unless it found totalShards
distinct (server, disk_id) targets, so a cluster with fewer disks than
shards (e.g. 8 single-disk servers for a 10+4 scheme) could never encode.
A disk safely holds several distinct shards of one volume: each is its own
.ecNN file and ReceiveFile keys by that extension. Drop the strict check and
let createECTargets round-robin shards across the available disks, matching
ec.encode's "4,4,3,3" fallback. The minTotalDisks floor (ceil(total/parity))
already keeps any disk under parityShards shards, so the volume still
survives losing any one disk.
Reserve capacity for the actual per-disk shard count rather than assuming
one shard each, so packing doesn't over-commit disk slots.
* fix(filer.sync): keep sync_offset fresh while the source is read-only
sync_offset holds the timestamp of the last replicated source event, so
monitoring derives lag from now-sync_offset. A read-only source emits no
metadata events, so the gauge froze at the last write and the derived lag
grew without bound, making thresholds unusable.
The source filer now sends an idle heartbeat carrying its current time
while a subscriber is caught up to the buffer head. filer.sync uses it to
advance the gauge, so now-sync_offset reflects real lag. Heartbeats are
opt-in (client_supports_idle_heartbeat), are never written to the metadata
log, and do not move the resume checkpoint, so a restart still resumes
from the last real event.
* fix(filer.sync): gate idle heartbeat on the read cursor, not SinceNs
In metadata-chunks mode persisted entries replay as log file refs and
never reach eachLogEntryFn, so lastSeenTsNs stays put and a caught-up
subscriber with an old SinceNs would never get a heartbeat. Use the
read cursor (lastReadTime), which advances in that mode too, max'd with
lastSeenTsNs so the in-memory backlog-then-idle case still works while
the cursor returned to the caller has not yet updated.
adjustAfterMove now removes the moved volume from the source disk's
VolumeInfos in place: it swaps the entry with the last one and nils the
tail. evacuateNormalVolumes ranges directly over that same slice, so the
niled tail slot is later read as a nil *VolumeInformationMessage and the
move attempt panics on vol.DiskType.
Iterate over a snapshot of the slice so in-place removals during a move
cannot leave nil holes in the loop.
Authentication records the identity with r.WithContext, which returns a
request copy. Handlers that log their own audit entry (PUT, DELETE,
tagging) see it, but GET/HEAD object and IAM operations rely on track()'s
fallback entry, which is built from the original request the auth copy
never reached - so requester came out empty.
Install a mutable identity holder on the request before authentication
and have SetIdentityNameInContext record into it. The holder is shared by
pointer across every request copy, so the fallback entry recovers the
authenticated requester. The per-request context value still takes
precedence, so nothing changes for handlers that see the auth copy.
* fix(shell): volume.balance no longer drains all volumes onto one server
The density-based capacity function reads per-disk VolumeInfos sizes, but
adjustAfterMove only updated VolumeCount and the selectedVolumes map. The
planner re-read a stale topology after every move, so the source node's
density never dropped and it kept moving volumes until that node was empty.
Move the volume's size accounting between disks after each planned move so the
density recomputes and the loop converges to an even distribution.
* refactor(shell): O(1) volume removal and direct disk lookup in adjustAfterMove
removeVolumeInfo swaps with the last element instead of shifting, and the disk
is fetched by key rather than ranging the DiskInfos map.
* fix(mount): keep periodic metadata flush from dropping concurrent chunk uploads
The periodic flush snapshotted entry.Chunks, then ran CompactFileChunks and
MaybeManifestize (the manifest upload is a network round trip) before
reassigning entry.Chunks. Async uploaders append freshly uploaded chunks
during that window, and the reassignment overwrote them: the data stayed on
the volumes but the file lost those chunk references, leaving zero-filled
holes on read. Large sequential writes such as cat of two 15 GiB files hit
several flush cycles and ended up corrupted.
Snapshot the chunk list under the entry lock with a length marker, do the
slow compaction and manifestization on the snapshot, then splice the
processed prefix back in front of whatever chunks arrived after the
snapshot.
* mount: drop redundant slice copies in the flush splice
processedPrefix is freshly built and the tail sub-slice is consumed
immediately under the entry lock, so append straight onto processedPrefix
instead of allocating two throwaway copies.
* chore(weed/command): prune unused functions
* drop now-unused closed field and renderLocked guard
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat(s3): add TagUser, UntagUser, ListUserTags IAM actions
Adds AWS IAM-compatible user tag operations on the embedded IAM
endpoint. Tags persist in the Identity proto as a repeated UserTag
field; the existing 50-tag / 128-byte-key / 256-byte-value AWS limits
are enforced. Pagination is stubbed (IsTruncated=false) since the
50-tag cap means all tags fit in a single response.
* review: validate UntagUser TagKeys entries
parseTagKeysParams now rejects empty keys and keys past
MaxUserTagKeyLength; UntagUser additionally requires at least one
TagKeys.member.N entry to match AWS validation behavior.
* review: pre-allocate user-tag merge and filter slices
mergeUserTags now allocates the combined existing+incoming capacity
up front; UntagUser builds the filtered slice via make with the full
ident.Tags capacity instead of ident.Tags[:0:0], which forced a
reallocation on every append.
* review: cover duplicate-in-request and invalid TagKeys cases
Regression tests assert TagUser rejects two members with the same key
in one request, and UntagUser rejects missing/empty/oversized TagKeys
entries.
* feat(s3): add four bucket configuration handlers
- GetBucketPolicyStatus: computes IsPublic from the existing bucket policy
- PutBucketRequestPayment: companion writer to the existing GET; accepts
only BucketOwner
- GetBucketAccelerateConfiguration: returns <Status>Suspended</Status>
- GetBucketLogging: returns an empty BucketLoggingStatus
Lets AWS SDK probes succeed instead of returning MethodNotAllowed.
* review: route GetBucketPolicyStatus through checkBucket
Mirrors the existence/auth gating used by other bucket handlers and
drops the bespoke filer_pb lookup so NoSuchBucket precedence is
consistent across the API surface.
* review: cap PutBucketRequestPayment body with MaxBytesReader
The body is unmarshalled as RequestPaymentConfiguration, which is a
handful of bytes; reject excessively large payloads up front and
defer Close immediately after wrapping.
* review: gate static getters on checkBucket
GetBucketAccelerateConfiguration and GetBucketLogging now run the
standard bucket existence check before returning the static
Suspended / empty-status response so a missing bucket cannot appear
to have valid configuration.
* review: share cache helper across misc tests; check io.ReadAll error
Accelerate and Logging tests now run through newMiscTestServer like
the others so the checkBucket guard sees a cached bucket; the
ReadAll error is explicitly checked.
* feat(s3): stub bucket configuration list endpoints
Adds Get and List handlers for Analytics, Inventory, IntelligentTiering,
and Metrics bucket configurations. List returns an empty result with
IsTruncated=false; single-get returns NoSuchConfiguration so SDK error
parsing remains predictable.
* review: gate stubs on bucket existence
All eight stub handlers now call checkBucket via stubBucketGuard so
NoSuchBucket takes precedence over NoSuchConfiguration / empty-list
responses, matching AWS S3 precedence. Tests provide a cached bucket
so the guard sees it as present.
* test(s3api): cover IAM inline policy aws:SourceIp + group inline gap
Unit tests under weed/s3api/ drive PutUserPolicy / PutGroupPolicy → reload
→ VerifyActionPermission with a synthetic 127.0.0.1 request and assert that
the policy's IpAddress condition flips the outcome.
The user-policy cases pass on master (hydrateRuntimePolicies already routes
inline docs through the policy engine, so Condition blocks are honored end-
to-end). The group-policy case fails: PutGroupPolicy still returns
NotImplemented, so a group inline doc never lands in the engine.
Integration counterparts live under test/s3/iam/ and exercise the same
paths against a live SeaweedFS S3+IAM endpoint.
* s3api: support group inline policies + Condition enforcement
PutGroupPolicy/GetGroupPolicy/DeleteGroupPolicy/ListGroupPolicies used to
return NotImplemented in embedded IAM mode, so anything attached to a
group as an inline doc — including aws:SourceIp or any other Condition —
was simply unreachable.
Wire the four endpoints to the credential-store methods that were
already in place (memory, postgres, filer_etc all implement
GroupInlinePolicyStore). On every config reload, hydrateRuntimePolicies
now also walks LoadGroupInlinePolicies, registers each doc in the IAM
policy engine under __inline_group_policy__/<group>/<policy>, and
appends that key to Group.PolicyNames so evaluateIAMPolicies picks it up
through its existing group walk. PutGroupPolicy/DeleteGroupPolicy are
added to the ReloadConfiguration trigger list in DoActions.
Side fix: MemoryStore.LoadConfiguration now surfaces store.groups too.
Without it iam.groups never repopulated on a memory-store reload, so
group policy evaluation silently no-op'd whether the policy was inline
or attached. The existing tests didn't notice because no test reloaded
through cm after creating a group.
The NotImplemented unit test is inverted to drive the new round-trip.
* s3api: drop redundant refreshIAMConfiguration from Put/DeleteGroupPolicy
DoActions already triggers ReloadConfiguration for both actions via the
explicit reload list, so calling refreshIAMConfiguration inline runs the
load twice per request. Per PR review.
* s3api: scope group-policy resource names per test; tighten deny polling
- Integration test resource names get a per-test suffix so retried or
parallel CI jobs don't trip EntityAlreadyExists / BucketAlreadyExists.
- Deny-path Eventually loops gate on AccessDenied via a typed helper
rather than any non-nil error; transient setup errors no longer end
the wait prematurely.
- ListGroupPolicies returns ServiceFailure when the credential manager
is nil, matching Put/Get/DeleteGroupPolicy.
* test(s3 iam): cover both IPv4 and IPv6 loopback in allow CIDRs
CI runners with happy-eyeballs resolve `localhost` to ::1 first, in
which case a 127.0.0.0/8-only allow would silently never match and the
deny-driven enforcement test would hang for the allow case. Add ::1/128
to every loopback-matching policy so the allow path works regardless of
which loopback family the SDK lands on.
* fix(ec): VolumeEcShardsInfo walks every disk on multi-disk servers
When a volume server holds EC shards for the same vid across more than
one disk, each DiskLocation registers its own EcVolume entry and
Store.FindEcVolume returns whichever one it hits first. The shard-info
RPC iterated only that single EcVolume's Shards, so the response missed
every shard mounted on a sibling disk.
The worker's verifyEcShardsBeforeDelete sums the per-server responses
into a union bitmap and refuses to delete the source volume when the
union falls short of dataShards+parityShards. On multi-disk
destinations, the union was systematically under-counted and source
deletion got blocked even though all shards were physically present and
mounted.
Walk every DiskLocation in the handler and emit the deduplicated union
of all shards. The .ecx-backed fields (file counts, volume size) still
come from a single EcVolume since every disk's entry opens the same
.ecx via NewEcVolume's cross-disk fallback.
Tests:
- TestVolumeEcShardsInfo_AggregatesAcrossDisks unit test in
weed/server/.
- test/volume_server/grpc/ec_verify_multi_disk_test.go integration test
drives the full generate -> mount -> redistribute -> restart ->
reconcile path and asserts both VolumeEcShardsInfo and
VerifyShardsAcrossServers + RequireFullShardSet (the production
source-deletion gate) report all 14 shards.
- ec_multi_disk_lifecycle_test.go tightened: replaces the
"VolumeEcShardsInfo only sees one disk's EcVolume" workaround with a
full-shard-set assertion.
* review: use ShardBits bitmask + cap-pre-allocation for shard dedup
* fix(s3): stop S3 Tables routes from swallowing buckets named "buckets" or "get-table"
The S3 Tables REST endpoints share top-level paths with the regular S3
API (/buckets for ListTableBuckets/CreateTableBucket, /get-table for
GetTable). They are registered first on the same router as the bucket
subrouter, so a path-style request such as GET /buckets?list-type=2 on
a bucket actually named "buckets" matched ListTableBuckets and returned
JSON. AWS SDK V2 (and Hadoop s3a / Spark) then failed XML parsing with
"Unexpected character '{' (code 123) in prolog".
Disambiguate by requiring the AWS V4 credential scope to name the
s3tables service on the colliding routes. Regular S3 SDKs sign with
service=s3, S3 Tables SDKs sign with service=s3tables, and the scope is
present in both the Authorization header and the X-Amz-Credential query
parameter for presigned URLs, so the matcher works for both flavors.
ARN-bearing S3 Tables routes (/buckets/<arn>, /namespaces/<arn>, etc.)
already cannot collide because colons are not valid in bucket names, so
they are left untouched.
* fix(s3): accept AWS JSON RPC content type as S3 Tables intent signal
The Iceberg catalog integration tests send unsigned PUT /buckets with
Content-Type: application/x-amz-json-1.1 to create table buckets. With
only the credential-scope check, those requests fell through to the
regular S3 CreateBucket handler and the suite went red on this branch.
Extend the matcher so a request is recognized as S3 Tables when either:
- its AWS V4 credential scope names SERVICE=s3tables; or
- it carries the canonical AWS JSON RPC 1.1 content type and is
unsigned (a request explicitly signed for SERVICE=s3 still wins).
The regular S3 SDKs do not send application/x-amz-json-1.1, so the
signal is safe for the colliding paths (/buckets, /get-table).
Also add an AWS SDK V2 for Go integration test under
test/s3/sdk_v2_routing/ that drives the SDK's own XML deserializer
against a bucket literally named "buckets" and "get-table" — the SDK
errors before the test asserts if the server returns the wrong body
shape. Wired up via .github/workflows/s3-sdk-v2-routing-tests.yml,
mirroring the etag/acl workflow.
* s3api: extend service matcher to all S3 Tables routes; simplify scope check
- Apply serviceMatcher to every S3 Tables route, not just the bare-path
ones. ARN-bearing paths could otherwise be hit by an S3 object key
that starts with arn:aws:s3tables:..., inside a bucket named
"buckets", "namespaces", "tables", or "tag". One matcher everywhere
closes both collision classes.
- Replace strings.Split + index lookup with strings.Contains for the
credential-scope check. The scope shape is fixed at
AK/DATE/REGION/SERVICE/aws4_request, slashes only delimit components,
and access keys are alphanumeric — so /s3tables/ matches iff SERVICE
is exactly s3tables. Existing unit cases (including the
access-key-substring case) still pass.
- Read the GetObject body in the SDK v2 routing test with io.ReadAll;
the single Read could return short and make the equality check flaky.
* s3api: drop content-type fallback; sign s3 tables harness traffic instead
The content-type fallback in isS3TablesSignedRequest let an anonymous
regular-S3 request whose body type is application/x-amz-json-1.1 hit
an S3 Tables route when the path-style object key happened to be
shaped like an S3 Tables ARN (e.g. PutObject on bucket "buckets"
with key arn:aws:s3tables:.../bucket/foo/policy). Narrow the matcher
back to the AWS V4 credential scope so only requests signed for
SERVICE=s3tables match the S3 Tables routes.
Update the Iceberg catalog test harness — the only caller still
sending unsigned PUT /buckets — to sign with SERVICE=s3tables. The
mini instance runs in default-allow mode, so the signature itself is
not verified; only the credential scope matters for the route match.
Drop the stale unit cases for the JSON-RPC content-type signal and
the routing test that exercised unsigned harness traffic.
* fix(volume): pass on-disk tombstone size to ReadData in verifyDeletedNeedleIntegrity
verifyDeletedNeedleIntegrity was forwarding TombstoneFileSize (-1) into
Needle.ReadData. A deletion tombstone is appended to .dat with DataSize=0
so the on-disk needle header carries Size=0; TombstoneFileSize is only
the .idx sentinel for "this entry is deleted" and is never written into
a needle header.
ReadBytes' size check therefore mismatched on every tombstone
(-1 != 0), returned ErrorSizeMismatch, and triggered the
4-byte-offset wrap-around retry in ReadData (offset + 32 GB). On any
volume large enough that offset+32 GB exceeds dat fileSize the retry
read EOF, CheckVolumeDataIntegrity reported corruption, and the loader
set noWriteOrDelete = true. Every volume whose last 10 .idx entries
included a deletion went read-only on startup — i.e. any healthy
volume where the most recent operations included a delete.
Pass Size(0) so the size check matches the on-disk tombstone header.
Add a regression test that writes three needles, deletes one, and
asserts CheckVolumeDataIntegrity succeeds with a tombstone at the .idx
tail. Without this fix the test reproduces the exact log shape from
the bug report:
read 0 dataSize 32 offset <orig+32GB> fileSize <much smaller>: EOF
verifyDeletedNeedleIntegrity ...idx failed: read data [N,N+32) : EOF
The Rust port guards its integrity-check size comparison with
!size.is_deleted() (seaweed-volume/src/storage/volume.rs) and never
hits this path, so no Rust mirror change is needed.
* test(seaweed-volume): mirror Go regression for deletion-tombstone integrity
The Rust integrity check already guards its size-mismatch comparison
with !size.is_deleted() (volume.rs:1859) and reads tombstone AppendAtNs
with body_size=0, so the Go regression fixed in the previous commit
does not apply. Lock that guarantee in with a parallel reload test:
write three needles, delete one, sync, reopen via Volume::new, assert
the volume is not flipped read-only.
Catches any future change that removes the deleted-entry guard or
re-introduces a size-strict path in check_volume_data_integrity for
tombstones.
* fix(volume): propagate io.EOF and ErrorSizeMismatch from verifyDeletedNeedleIntegrity
CheckVolumeDataIntegrity relies on identity comparison against io.EOF
and ErrorSizeMismatch to walk back through the last ten .idx entries
and tolerate a partial truncation at the tail (the "fix and continue"
loop). The live-needle branch in doCheckAndFixVolumeData already
returns those sentinels unwrapped; the deletion branch wrapped them
in fmt.Errorf, so a genuine .dat truncation past a tombstone offset
broke the recovery and flipped the volume read-only.
Mirror the live-needle handling: both verifyDeletedNeedleIntegrity
and doCheckAndFixVolumeData now short-circuit on io.EOF /
ErrorSizeMismatch and pass them through unwrapped. Other errors keep
their existing context wrapping.
Also tighten the regression test to capture lastAppendAtNs and assert
it's non-zero, so a future regression that skips the tombstone body
(and therefore never populates AppendAtNs) is caught even when the
err check still passes.
fix(s3): keep anonymous access working with EnableIam default
`docker run seaweedfs` (and `weed mini` with no config) start with
EnableIam=true but no IAM config file and no identities. The advanced-IAM
init path was failing in 4.25 because of the missing STS signing key,
which masked a latent bug: SetIAMIntegration unconditionally flipped
isAuthEnabled to true, and isEnabled() also treated a non-nil
iamIntegration as auth-on. Once the mini SSE-S3 KEK landed in 4.26 the
STS fallback started succeeding, the integration got installed end to
end, and every anonymous S3 request bounced as AccessDenied.
Separate the two concerns: SetIAMIntegration just plumbs in the OIDC /
embedded-IAM machinery, and a new EnableAuthEnforcement opts in to
enforcement. The startup path calls it only when -s3.iam.config is
actually provided, so operators with explicit IAM configs still get auth
(preserves #7726). isEnabled() now reads isAuthEnabled only.
* feat(filer): add atime field and TouchAccessTime RPC to filer proto
Introduce POSIX-style access-time tracking on the filer:
- FuseAttributes gains atime (field 22) and atime_ns (field 23).
- New TouchAccessTime RPC (and Touch{Access,Time}{Request,Response})
lets read paths bump atime without going through UpdateEntry's
chunk-rewrite/EqualEntry short-circuit.
Additive proto changes only; zero atime is treated as unset and
existing clients are unaffected. Java client proto is kept in lock
step.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(filer): wire Atime through Attr codec with mtime fallback
Add Attr.Atime and round-trip it through EntryAttributeToPb /
EntryAttributeToExistingPb / PbToEntryAttribute. A zero proto atime
decodes as Mtime, so legacy entries report a sensible value and
freshly-created/updated entries default Atime to Mtime when callers
do not set it explicitly.
CreateEntry and UpdateEntry stamp Atime = Mtime (or Crtime) when it
is zero. TouchAccessTime later bypasses this path to write atime
alone via Store.UpdateEntry.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(filer): preserve atime in first epoch second on decode
The Atime decode branch previously treated any attr.Atime == 0 as
unset and overwrote it with Mtime, which drops valid timestamps in
the first second of the unix epoch where attr.Atime is 0 but
attr.AtimeNs > 0. Check both fields so we only fall back to Mtime
when both are zero.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Volumes written by versions before 3.09 (commit 056c480eb) store the
needle checksum using the deprecated CRC.Value() transform. When the
read path moved into readNeedleTail, the fallback that accepts both
encodings was dropped, so .dat files copied from old installs now fail
verification with "invalid CRC ... data on disk corrupted" even though
the data is intact. Restore the dual check, matching the surviving
fallback in volume_read.go.
* s3api: verify source permission on CopyObject and UploadPartCopy
The Auth middleware only authorized the destination because routes key on
the request URL. The source from X-Amz-Copy-Source was never evaluated,
so an STS session token scoped to one prefix could copy from any other
prefix in the same bucket.
Add AuthorizeCopySource on IdentityAccessManagement to run the full
bucket-policy + IAM/identity flow against the source, using a synthetic
GetObject request so action resolution lands on s3:GetObject (or
s3:GetObjectVersion when a source versionId is supplied). Both
CopyObjectHandler and CopyObjectPartHandler now invoke it before reading
the source.
* s3api: preserve presigned-URL session token on copy-source check
Presigned CopyObject / UploadPartCopy requests carry the STS session
token in the query string (X-Amz-Security-Token), not in a header.
Rebuilding the synthetic source URL from scratch dropped that token, so
the source authorization would fall through to non-STS paths and miss
session policy enforcement. Forward X-Amz-Security-Token from the
original query (alongside versionId), still excluding unrelated params
like uploadId/partNumber that would steer ResolveS3Action away from
s3:GetObject.
* fix(volume): reopen .idx writable after MarkVolumeWritable
When .vif has ReadOnly=true, load() opens .idx as O_RDONLY and builds a
SortedFileNeedleMap whose Put returns os.ErrInvalid. MarkVolumeWritable
only flipped noWriteOrDelete back to false and rewrote .vif, so writes
still failed at v.nm.Put. Reopen .idx in O_RDWR and rebuild v.nm in its
writable form (in-memory or leveldb small/medium/large) before flipping
the flag.
Mirror the same fix in seaweed-volume: the Rust load path leaves
CompactNeedleMap/RedbNeedleMap with no idx_file writer when the volume
boots read-only, so post-MarkVolumeWritable puts silently succeeded
in-memory only and were lost on the next restart. set_writable now
reattaches an append-mode writer when one is missing.
* fix(volume): keep old needle map until replacement is built; defer writable flag
Go: build the writable needle map into a local before swapping. A
construction failure now leaves v.nm pointing at the original
SortedFileNeedleMap so MarkVolumeWritable can roll back, instead of
stranding the volume with v.nm == nil.
Rust: attach the .idx writer before flipping no_write_or_delete to
false. A transient open/metadata failure used to leave the volume
marked writable with no writer attached, and subsequent puts would
silently skip the on-disk append.
* fix(admin): switch file browser upload/download to filer gRPC + volume HTTP
The admin file browser proxied uploads and downloads through the filer's
HTTP listener, so the whole feature 404'd against filers started with
-disableHttp=true even though S3 still worked on its own port. Re-route
through the filer gRPC service: LookupDirectoryEntry + StreamContent for
reads (chunks flow straight from the volume servers), AssignVolume +
volume HTTP POST + CreateEntry for writes. Volume read tokens come from
jwt.signing.read.key when configured; the old jwt.filer_signing tokens
no longer apply since the filer HTTP surface is bypassed.
* admin file browser: propagate request context + track response writes
Pass r.Context() into uploadFileToFiler so a client disconnect cancels
the in-flight chunked upload instead of letting it run to completion
against the volume servers. For DownloadFile, replace the Content-Type
probe with a small response-writer wrapper that records whether headers
or bytes have actually been sent, so the error path can't silently
convert a pre-stream failure into a partial response if future code
moves the header-setting around.
When s3.domainName is set, all bucket-prefix routes were gated on a
matching Host header. Requests that arrive via an IP, an unlisted
hostname, or a reverse proxy that rewrites Host hit no router and bounce
back as 405/404 (and 503 once a proxy maps the upstream error).
Register the path-style catch-all unconditionally, after the
host-specific routers, so it only fires when no Host matcher applies.
TaskSource and TaskTarget carry disk_id on the wire, but the execution
plan map built for the admin UI dropped the field entirely. On a
multi-disk node holding shards of the same volume, there was no way to
tell from the plan which disk would receive each shard. Include
disk_id on each endpoint and target_disk_id on each shard assignment,
and extend the existing execution-plan test to set and assert the
field.
Updated the README to reflect the current status of MinIO, noting its ceased development and security concerns, along with changes in the descriptions of its features compared to SeaweedFS.
* fix(ec_distribute): remove partial files on copy stream error
writeToFile opens the destination with O_TRUNC and streams into it. On
a mid-stream receive / write / cancellation error it returned the
failure but left the destination behind in whatever state had been
written so far — typically 0 bytes when the source errored before
sending any FileContent. VolumeEcShardsCopy distributes .ecx by
calling doCopyFile, so this same stub-leaving behaviour produced the
0-byte .ecx files seen on EC encoding failures: the source claims a
non-zero ModifiedTsNs (so the existing "source not found" cleanup
doesn't fire), the stream then errors immediately, and the receiver
ends up with a 0-byte .ecx that downstream code mistook for a valid
empty index.
Clean up the partial file on every error path that returns from the
streaming loop (receive, write, and cancellation). Skip cleanup when
isAppend=true so resumable appends keep their existing content. As
defense in depth, VolumeEcShardsCopy also stats the .ecx after copy
and removes / errors on a 0-byte result so the orchestrator can pick
a different source.
The Rust volume server has only the source side of CopyFile (no
client-side stream-to-disk consumer) and no .ecx subsystem yet, so
this fix has no Rust mirror.
* fix(ec_distribute): close file before remove, fail fast on stat error
Address review feedback:
- writeToFile's mid-stream removeIncomplete called os.Remove while the
destination file handle was still open. On Windows os.Remove fails
while a handle is open, so the cleanup wouldn't run there. Wrap the
handle close in a once-only helper, call it from removeIncomplete
and from the existing "source not found" cleanup, and keep a deferred
close as the safety net for the normal-return path.
- VolumeEcShardsCopy's post-copy .ecx check silently passed when
os.Stat returned an error: doCopyFile had reported success but if
the file was already gone, unreadable, or somehow a directory, the
orchestrator only learned at mount time with no useful context.
Treat any non-nil stat error and any directory result as a copy
failure here and surface it immediately.
* fix(ec_mount): reject 0-byte .ecx and aggregate cross-disk failures
MountEcShards's per-disk loop bailed on the first disk returning a
non-ENOENT error, and NewEcVolume wrapped its ENOENT with %v so the
caller's `err == os.ErrNotExist` check never matched. On a multi-disk
volume server where ec.balance / ec.rebuild had distributed shards
across sibling disks while the matching .ecx never arrived, the mount
loop bailed after disk 0 with "cannot open ec volume index" and the
operator never saw that the rest of the disks were also empty. The
companion failure mode is a 0-byte .ecx stub left by EC distribute's
writeToFile after a mid-stream copy failure: Stat() succeeds, treating
the stub as a valid index, and downstream mount work proceeds against
an empty file.
Wrap the ec-volume open errors with %w, treat a 0-byte .ecx as
os.ErrNotExist (in NewEcVolume, findEcxIdxDirForVolume, and
HasEcxFileOnDisk), and have MountEcShards collect per-disk failures
before returning a single aggregated error. The "no .ecx anywhere"
case gets a distinct error so the orchestrator can re-copy the index
from a healthy replica rather than retry against the same broken
state.
* fix(ec_reconcile): indexEcxOwners also rejects 0-byte .ecx stubs
findEcxIdxDirForVolume already skipped 0-byte .ecx during MountEcShards,
but indexEcxOwners (used by reconcileEcShardsAcrossDisks at startup)
still recorded the first .ecx by name only. On a store where one disk
holds a 0-byte stub left by a failed EC distribute and a sibling disk
holds the real index, the stub would win the owner selection — and
NewEcVolume's new size check would then refuse to load against it,
leaving the orphan shards unloaded even though a valid index exists.
Mirror the size check from findEcxIdxDirForVolume: skip directory
entries whose .ecx Info() reports size 0 or whose Info() call fails.
* fix(ec_mount): accept 0-byte .ecx as valid empty index
The previous commit treated a 0-byte .ecx in NewEcVolume as
os.ErrNotExist, on the assumption that any empty .ecx was a stub left
by a failed copy stream. That broke the legitimate empty-volume case:
when an EC volume's source .idx has no live entries (e.g. all needles
deleted before WriteSortedFileFromIdx), the sorted .ecx is genuinely
0 bytes and must mount. The integration test
TestEcShardsToVolumeMissingShardAndNoLiveEntries fails with
"MountEcShards: no .ecx index found on any local disk" because the
mount path now refuses the legitimate empty index.
A 0-byte .ecx left by a failed copy stream is indistinguishable from
the legitimate empty case by file size alone. Preventing stub files
from being written is the receiver-side cleanup in writeToFile's job
(the companion EC distribute PR), not NewEcVolume's at mount time.
The cross-disk lookup helpers (findEcxIdxDirForVolume, HasEcxFileOnDisk,
indexEcxOwners) keep their size > 0 preference: when a real .ecx
exists on a sibling disk alongside a stub, we still want to route
mounts and reconcile at the real one. If no non-zero .ecx exists
anywhere, the per-disk fallback in MountEcShards can still open the
0-byte .ecx and the volume mounts.
Replace TestMountEcShards_ZeroByteEcxOnlyDisk with
TestMountEcShards_EmptyEcxMountsSuccessfully, which pins the
empty-volume invariant.
* fix(volume.list): show one entry per physical disk on multi-disk nodes
DataNodeInfo.DiskInfos is keyed by disk type, so several same-type
physical disks on one node collapse to a single map entry at the master.
volume.list iterated that map directly and reported one "Disk hdd ...
id:0" line per node, hiding the per-disk volume and shard layout. EC
operators on multi-disk volume servers had no way to verify which
physical disk a shard landed on.
Lift the per-physical-disk split into a DiskInfo.SplitByPhysicalDisk()
method on the proto type so consumers outside admin/topology can use
it. Apply it in writeDataNodeInfo so the verbose Disk block shows one
entry per physical disk, ordered by DiskId. Capacity counters are
split evenly across reconstructed disks since the wire format doesn't
carry per-disk capacity yet.
This is a display-only change. ActiveTopology already did the split on
its own and is now updated to call the shared helper.
* fix(volume.list): preserve totals, count active/remote exactly, dedupe header
Address review feedback on the per-physical-disk split:
- share() truncated remainders so reconstructed per-disk counters could
sum to less than the original aggregate (10 / 3 = 3+3+3). Distribute
the remainder to the lowest disk ids so MaxVolumeCount and
FreeVolumeCount sum exactly back to the node totals.
- ActiveVolumeCount and RemoteVolumeCount are derivable per disk from
the VolumeInfos already grouped by DiskId, so count them exactly
(ReadOnly=false and RemoteStorageName!="" respectively) instead of
approximating with an even split.
- writeDataNodeInfo's per-disk callback fired the DataNode header on
every iteration after the split, so a node with 6 physical disks
emitted 6 DataNode headers. Guard the callback with headerPrinted so
the header still appears at most once per node.
- Sort split disks deterministically using explicit DiskId comparison
to avoid int overflow risk on 32-bit systems.
- Tighten the volume.list test substring to "id:N\n" so unrelated
tokens like "ec volume id:101" don't accidentally match the id:1
needle, and assert the rack callback fires once.
The 127.0.0.1-only reservation in AllocateMiniPorts/AllocatePortSet let
another process hold the gRPC port on a different interface, so weed
mini's isPortAvailable check failed and it shifted master.grpc. weed
shell -master=<HTTP> still derives grpc as HTTP+10000 and dialed the
unused port, hanging until the 30s context deadline killed it. Bind the
reservation listeners on :port to match mini's check.
Also bound listFilerContents in catalog_risingwave with a 30s
exec.CommandContext so a hung weed shell during failure-cleanup can't
burn the 20-minute test budget.
When jwt.filer_signing.key is set, the filer's IamGrpcServer requires
a Bearer token on every IAM RPC. The shell's s3.* IAM commands dialed
without that header and failed with Unauthenticated. Route them through
a small helper that mints a token from the same key viper-loaded from
security.toml and appends it as outgoing metadata, matching the credential
grpc_store pattern.
When weed filer started its embedded S3 gateway with -s3 -s3.config, only
the S3 server loaded the s3.json static identities — the filer's own
CredentialManager stayed empty, so the IAM gRPC service backing the admin
UI and weed shell returned only dynamic users. Mirror the wiring weed
server already does and hand the same config path to the filer.
fix(mount): don't release file handles from Forget
Forget(nodeid, nlookup) only decrements the kernel inode lookup count.
File handle lifecycle belongs to FUSE Open/Release. Driving the FH
refcount from Forget coupled two unrelated counters and could tear down
a still-live handle if Forget ever raced ahead of Release.
Drop the ReleaseByInode call (and the now-unused method).
2026-05-18 01:02:58 -07:00
1602 changed files with 182488 additions and 35184 deletions
org.opencontainers.image.description=SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast!
if (Test-Path S:\walk) { throw "directory survived recursive delete" }
Write-Host "::endgroup::"
- name:Logs
if:always()
shell:pwsh
run:|
foreach ($f in 'C:\seaweed-mount.log','C:\seaweed-mount.err.log','C:\seaweed-remount.log','C:\seaweed-remount.err.log','C:\seaweed-remount2.log','C:\seaweed-remount2.err.log','C:\seaweed-dirmount.log','C:\seaweed-dirmount.err.log','C:\seaweed-mini.log','C:\seaweed-mini.err.log') {
echo "- Suite: test/volume_server/${{ matrix.test-type }} (shard ${{ matrix.shard }} of ${SHARD_COUNT}, see 'Select this shard's tests' for the split)" >> "$GITHUB_STEP_SUMMARY"
make test-with-server TEST_PATTERN="TestVersioningCompleteMultipartUploadIsIdempotent|TestVersioningSelfCopyMetadataReplaceCreatesNewVersion|TestVersioningSelfCopyMetadataReplaceSuspendedKeepsNullVersion|TestSuspendedDeleteCreatesDeleteMarker"
# Run every versioning test, so a regression test lands covered instead
# of waiting for someone to remember this file. Name a test in EXCLUDE,
# with the reason, to keep it out.
#
# TestVersioningPagination*: opt-in stress tests that build 1500+
# versions. They self-skip without ENABLE_STRESS_TESTS and have their
# own make target, so this gate should not carry them.
echo "- Command: go test -v -count=1 -timeout=${{ env.TEST_TIMEOUT }} ./test/volume_server/${{ matrix.test-type }}/... -run \"${TEST_PATTERN}\"" >> "$GITHUB_STEP_SUMMARY"
echo "- Suite: test/volume_server/${{ matrix.test-type }} (shard ${{ matrix.shard }} of ${SHARD_COUNT}, see 'Select this shard's tests' for the split)" >> "$GITHUB_STEP_SUMMARY"
echo "- Command: go test -v -count=1 -timeout=${{ env.TEST_TIMEOUT }} ./test/volume_server/${{ matrix.test-type }}/... -run \"\${TEST_PATTERN}\"" >> "$GITHUB_STEP_SUMMARY"
@@ -425,7 +426,7 @@ If the hot/warm data is split as 20/80, with 20 servers, you can achieve storage
## SeaweedFS Filer ##
Built on top of the blob store, SeaweedFS Filer adds directory structure to create a file system. The directory sturcture is an interface that is implemented in many key-value stores or databases.
Built on top of the blob store, SeaweedFS Filer adds directory structure to create a file system. The directory structure is an interface that is implemented in many key-value stores or databases.
The content of a file is mapped to one or many blobs, distributed to multiple volumes on multiple volume servers.
@@ -509,20 +510,22 @@ SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Sqlite, Mong
### Compared to MinIO ###
MinIO follows AWS S3 closely and is ideal for testing for S3 API. It has good UI, policies, versionings, etc. SeaweedFS is trying to catch up here. It is also possible to put MinIO as a gateway in front of SeaweedFS later.
Please note, as Apr 25, 2026 MinIO ceased development. It's strongly discouraged to use that unmaintained software with multiple security bugs.
MinIO metadata are in simple files. Each file write will incur extra writes to corresponding meta file.
MinIO followed AWS S3 closely and was ideal for testing for S3 API. It had good UI, policies, versionings, etc. SeaweedFS is trying to catch up here.
MinIO does not have optimization for lots of small files. The files are simply stored as is to local disks.
MinIO metadata were in simple files. Each file write will incur extra writes to corresponding meta file.
MinIO did not have optimization for lots of small files. The files were simply stored as is to local disks.
Plus the extra meta file and shards for erasure coding, it only amplifies the LOSF problem.
MinIO has multiple disk IO to read one file. SeaweedFS has O(1) disk reads, even for erasure coded files.
MinIO had multiple disk IO to read one file. SeaweedFS has O(1) disk reads, even for erasure coded files.
MinIO has full-time erasure coding. SeaweedFS uses replication on hot data for faster speed and optionally applies erasure coding on warm data.
MinIO had full-time erasure coding. SeaweedFS uses replication on hot data for faster speed and optionally applies erasure coding on warm data.
MinIO does not have POSIX-like API support.
MinIO did not have POSIX-like API support.
MinIO has specific requirements on storage layout. It is not flexible to adjust capacity. In SeaweedFS, just start one volume server pointing to the master. That's all.
MinIO had specific requirements on storage layout. It is not flexible to adjust capacity. In SeaweedFS, just start one volume server pointing to the master. That's all.
In a namespace with a default-deny policy the install hangs: the components cannot resolve each other, and the post-install bucket hook waits on the master and filer until it gives up. `networkPolicy.enabled` renders one `NetworkPolicy` per component, selecting its pods by the standard `app.kubernetes.io/{name,instance,component}` labels and admitting traffic from the other pods of the release on the ports that component listens on.
That alone leaves outbound traffic untouched, which is enough when the namespace's default-deny only restricts ingress. If it lists `Egress` in its `policyTypes` too - the usual baseline - the components still cannot resolve DNS, and you need the second opt-in below as well.
Egress is separate because the chart knows where its own components live but not where your filer store, notification sink or remote tier does, and because in a namespace with no default-deny at all, adding egress rules would narrow the components from "may reach anything" to "may reach these peers":
```yaml
networkPolicy:
enabled: true
egress:
enabled: true
kubeApiServer:
# the endpoint behind the kubernetes service, not its ClusterIP
cidrs: ["172.18.0.2/32"]
extraEgress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: postgresql
ports:
- protocol: TCP
port: 5432
```
`kubeApiServer.cidrs` is only demanded when something in the release actually needs the API server, which is the COSI sidecar and, on an upgrade that grows a volume PVC, the resize hook. No seaweedfs component itself speaks to it.
Anything reaching the release from outside - an ingress controller, a Prometheus in another namespace - goes into `networkPolicy.extraIngress`, or into `networkPolicy.components.<component>.extraIngress` for a single component. See the `networkPolicy` block in `values.yaml` for the full set.
Two things worth knowing before you turn this on:
- **Monitoring stops.** The metrics ports are admitted from release pods like every other port, so with `global.seaweedfs.monitoring.enabled` the ServiceMonitors keep scraping targets a Prometheus in another namespace can no longer reach. Nothing reports it; add the scraper's namespace to `extraIngress`.
- **The resize hook's policy is a Helm hook.** Its Job runs before the release manifest is applied, so the policy has to be a `pre-install` hook too. Helm does not garbage-collect hook resources, so on an upgrade that grows a volume PVC the policy is created and then left behind on uninstall - delete `<release>-seaweedfs-volume-resize-hook` by hand if it bothers you.
The DNS selectors default to CoreDNS as kubeadm, kind and the managed offerings from AWS, Google and Azure install it. On OpenShift, override `egress.dnsNamespaceSelector` and `egress.dnsPodSelector` to match `openshift-dns`; see the comment in `values.yaml`.
## OpenShift Support
SeaweedFS can be deployed on OpenShift or any cluster enforcing the Kubernetes "restricted" Pod Security Standard. By default, OpenShift blocks containers that run as root or use `hostPath` volumes.
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion }}
service:
name:{{$serviceName }}
port:
number:{{.Values.filer.grpcPort }}
{{- else }}
serviceName:{{$serviceName }}
servicePort:{{.Values.filer.grpcPort }}
{{- end }}
{{- end }}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Chris Lu
Chris LuandGitHub
Mohit TalniyaandGitHub
Dmitriy PavlovandGitHub
Lars LehtonenandGitHub
Andrei StroianandGitHub
github-actions[bot]
Lisandro PinandGitHub
Lukas KalliesandGitHub
Kobi HikriandGitHub
TJDawson10andGitHub
qzhelloandGitHub
Chris WydraandGitHub
jayandGitHub
Cole HelblingandGitHub
Peter DoddandGitHub
jk2lxandGitHub
os-pradipbabarandGitHub
patrickandGitHub
msementsovandGitHub
Bruce ZouandGitHub
Sergey ZinchenkoandGitHub
Minsoo KimandGitHub
Ruslan Bel'kovandGitHub
ahalaunandGitHub
ᎠᎡ. Ѕϵrgϵ ѴictorandGitHub