The max volume count subtracted each volume's remaining room from the
free disk space, except when the master had preallocation turned on,
in which case it subtracted nothing. That is wrong for a disk that
mixes preallocated and plain volumes, and on platforms where
preallocation is not supported.
Use the larger of the .dat file's size and its allocated blocks, in the
Go and the Rust volume server, so a preallocated volume counts as full
and a plain one keeps its remaining room reserved.
Fixes#5778
Co-authored-by: Dan Foster <4570306+danfoster@users.noreply.github.com>
* iceberg: keep a default-CRS geometry column without a CRS after compaction
Writers such as DuckDB write a GEOMETRY column whose Iceberg type has the
default CRS (OGC:CRS84) with no CRS in the Parquet logical type. parquet-go
spells the default out when it writes such a column (geometryType's
LogicalType), even when the column it read declared none, so every
compacted file declared GEOMETRY(crs=OGC:CRS84) where its inputs declared
GEOMETRY. Both mean the same to the Parquet spec, but a reader may treat the
file column as another type than the table column: DuckDB casts it to the
table's GEOMETRY, and a cast column is no longer pruned by its row-group
statistics, so a spatial filter read every row group of a compacted file.
After a bin is merged, the footer's schema elements of every GEOMETRY
column whose Iceberg type has the default CRS (top-level or nested) are
rewritten to declare no CRS. The data, the page indexes and every offset
stay byte for byte; only the footer is re-encoded. An explicit CRS is left
as it is. Retyping the writer's column instead does not work: parquet-go
refuses to convert between two GEOMETRY types that both mean the default,
and its sorting writer converts.
Tests: geometry_crs_test.go, a bin-packed and a sorted compaction of
default-CRS inputs (one of them spelling the default out) write no CRS,
with a control showing parquet-go spells it out on its own; an explicit CRS
is kept; a nested geometry. On master the binpack and sort cases fail with
CRS "OGC:CRS84".
* iceberg: compress the row lineage columns of compacted v3 files
A v3 compaction appends _row_id and _last_updated_sequence_number to the
merged file. The two leaves were built without a codec, so they were
written uncompressed, 16 bytes a row, whatever codec the data columns use;
on a table of points the compacted files grew by about a third.
The lineage leaves now take the codec of the reference file's first leaf
that declares one, ZSTD when none does.
Test: TestCompactCompressesLineageColumns; on master both columns are
written UNCOMPRESSED next to ZSTD data columns.
* iceberg: match list and map geometry by Iceberg name in files without field ids
A Parquet footer path runs through the repeated group of a LIST or MAP
(shapes.list.element, attrs.key_value.value), which Iceberg leaves out of
its field names (shapes.element, attrs.value). The name lookup for a file
without field ids therefore missed every geometry inside a list or a map
and left the default CRS spelled out. Name the paths as Iceberg does,
including the legacy two-level list forms.
* iceberg: rewrite the geometry CRS footer inside the merged file's buffer
Removing a CRS shortens the footer, so the new one fits where the old one
was. Copying the whole merged file to replace a few hundred bytes of
footer doubled the memory of a compaction near the target file size.
* iceberg: take the lineage codec from the data columns, not an input's _row_id
An input that already carries _row_id may list it before its data
columns, and write it uncompressed; the new lineage columns then copied
that codec instead of the data columns'. Choose from the columns the
layout keeps.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* s3tables: accept the external-volume verification object at a table bucket's root
Snowflake verifies an S3-compatible external volume before it creates or
writes Iceberg tables in it (SYSTEM$VERIFY_EXTERNAL_VOLUME, also run
implicitly): it writes, reads, lists and deletes one small object,
<base>/verify_<epoch milliseconds>_<number>, a 21-byte body. A table bucket
refuses every key outside the Iceberg layout, so the check failed and
Snowflake could not use a table bucket as an external volume.
PUT, GET, HEAD, DELETE and DeleteObjects now accept exactly that key in a
table bucket, narrowly enough that it cannot be used to store data:
- at the bucket root only, so nothing is encoded in a prefix, written into
a hidden tree or placed on a namespace or table name;
- one shape: verify_<13 digits of epoch ms within one hour of the server's
clock>_<1-10 digits>;
- a PUT must declare a Content-Length of at most 1 KiB (no chunked or
undeclared body) and carry no user metadata, tagging, Content-Disposition,
Cache-Control, Expires or website redirect.
Multipart, copy, rename, ACL, tagging and POST policy writes keep the plain
layout rule. Buckets that are not table buckets are unaffected.
Tests: table_bucket_verify_object_test.go, the key accepted (with the
Iceberg layout as the positive control and the plain rule still refusing
it), 15 look-alikes refused (prefixed, hidden tree, wrong shape, outside the
window, ordinary keys), and seven PUT refusals through PutObjectHandler.
* s3tables: keep the verification key off namespaces and narrow its PUT
A namespace may be named like a verification key (verify_<ms>_<n>), and
the key then names the namespace's directory. A PUT replaced that
directory with a prefix object, dropping the namespace's catalog
metadata; a DELETE removed an empty namespace. The key now gets the plain
layout rule when it names a directory or its entry cannot be read, and a
PUT of it never falls back to a prefix object, so a namespace created
after the check is not replaced either.
The time window applies to PUT only: GET, HEAD and DELETE take a key of
any date, so an object a check left behind for more than an hour can
still be read and removed. A verification PUT also refuses an extraction
request and the stored Content-Language, Response-Content-Disposition and
Content-Encoding headers (aws-chunked framing alone is still accepted).
The integration test runs the check's PUT, HEAD, GET, list and DELETE
against a table bucket, and the PUT and DELETE over a namespace of the
key's shape.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* s3 backend: optional HTTP client settings for the cloud tier
The S3 tier backend builds its client from a bare aws.Config, so it always
uses Go's default transport: HTTP/2 wherever the endpoint offers it over TLS,
no cap on connections per host, and a 90s idle timeout. Against some
S3-compatible services that is a poor fit. Over HTTP/2 every upload and
range read of a tiered volume shares one connection and its flow-control
window; with a small window, tier uploads and concurrent reads of tiered
volumes become throughput-bound on that one connection, while HTTP/1.1 opens
a connection per concurrent request. Some services also limit parallel
connections per client address, or drop idle connections before 90s.
Add three optional settings to [storage.backend.s3.<id>], all off by
default so existing deployments keep the SDK's default client:
- disable_http2: keep the client on HTTP/1.1 (ForceAttemptHTTP2=false and an
empty TLSNextProto, the documented opt-out);
- max_connections: MaxConnsPerHost, 0 = no cap;
- idle_connection_timeout_seconds: IdleConnTimeout, 0 = Go's default.
The session cache key includes the options when any is set, so two backends
on the same endpoint with different settings do not share a client.
ToProperties reports the settings; the master.toml scaffold documents them.
Tests: TestS3BackendHTTPOptionsReachTheClient fails without the change (the
settings never reach the client); TestS3BackendDefaultHTTPClientUnchanged,
TestS3BackendSessionsDifferByHTTPOptions, TestParseNonNegative.
* s3 backend: reject an idle timeout too large for time.Duration
An idle_connection_timeout_seconds above math.MaxInt64/1e9 overflowed when
multiplied by time.Second. The negative result made the transport keep its
default timeout and ToProperties export a negative value. Warn and use the
default instead.
* s3 backend: parse the idle timeout as 64-bit seconds
parseIdleConnTimeout went through strconv.Atoi, so on 32-bit platforms
any value above 2147483647 seconds was rejected while the same value was
accepted on 64-bit, and the overflow test failed under GOARCH=386. Parse
with ParseInt(…, 64) and export the effective value with FormatInt so the
accepted range is the same on every platform.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:44:39 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
s3api: exercise the inline-content route through CopyObjectPartHandler
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: answer InvalidBucketName to a denied request on a malformed bucket name
The store authorizes a request before it looks at the bucket, so a caller
denied on a bucket whose name breaks the S3 naming rules (upper case, an
underscore, too short, two adjacent periods, a non-ASCII letter) gets 403
AccessDenied, where AWS S3 answers 400 InvalidBucketName. Clients and
compatibility suites that check the error code read the 403 as a permission
problem.
Auth and AuthPostPolicy now turn an AccessDenied for a resolved identity into
InvalidBucketName when the bucket name fails VerifyS3BucketName or uses a
character outside a-z, 0-9, '.' and '-' (VerifyS3BucketName checks with
unicode.IsLower/IsNumber, which accept letters such as 'ü'). The answer depends
on the name alone, so it reveals nothing about which buckets exist. An
authentication failure, an allowed request and a denial on a well-formed name
answer as before.
Test: TestAuthDeniedInvalidBucketName drives iam.Auth with signed and anonymous
requests; on master every malformed name answers 403, with the change 400,
while the identity's own bucket stays allowed and a well-formed foreign name
stays 403. TestRefineDenialForBucketName covers the rule table.
* s3: refine POST-policy and deferred anonymous-read denials on a malformed bucket name
Two denial paths bypass the refinement in Auth and AuthPostPolicy:
- A browser POST upload carries its credentials in the form, so
AuthPostPolicy delegates it to PostPolicyBucketHandler, which verifies the
form and writes AccessDenied itself when the signer may not write the bucket
or the per-key authorization denies the object. Both denials now go through
refineDenialForBucketName with the verified identity.
The coarse Write check used to run inside doesPolicySignatureV2Match and
doesPolicySignatureV4Match before the signature was compared, so a form with
a wrong signature was told whether the key may write the bucket. It now runs
in doesPolicySignatureMatch after the signature verifies, as on the
header-signed path; a form that fails verification answers
SignatureDoesNotMatch (or MissingFields) and is never refined.
- Unsigned GET/HEAD object reads are deferred by AuthWithPublicRead to the
object handlers, where writeAnonymousAwareNotFound masks a miss as
AccessDenied. That denial is refined with the anonymous identity, so a
missing malformed bucket answers InvalidBucketName when an anonymous
identity is configured, as Auth does. A denial on an entry that was found
is left alone: the bucket exists.
Test: TestPostPolicyDeniedInvalidBucketName drives AuthPostPolicy with signed
forms (Write denied, per-key policy deny, bad signature, well-formed name);
TestAnonymousReadDeniedInvalidBucketName drives AuthWithPublicRead with GET and
HEAD, with and without an anonymous identity. Both fail before this change.
* s3: refine a denied anonymous read of an existing key on a malformed bucket name
A deferred anonymous GET/HEAD refined only its masked miss to
InvalidBucketName, so in a bucket whose name breaks the S3 rules (one
VerifyS3BucketName still accepts, such as a non-ASCII name) a missing key
answered 400 and an existing private key 403, telling the caller which
keys exist. The entry check now refines its denial the same way, so both
answer alike. Without an anonymous identity no identity resolves and both
stay 403, as in Auth.
The deny-precedence case in TestAnonymousObjectACLConfiguredIdentity runs
on the fixture bucket "b", shorter than S3's three-character minimum, so
its denial now answers InvalidBucketName.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:08:33 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An out-of-band empty directory listed under its own trailing-slash
prefix answered the probe with a fake entry of itself: a Contents key on
4.48 and, since recently, a CommonPrefix. That listing enumerates
children, where S3 returns nothing for an empty prefix, and clients that
read every returned key (Pulumi lock listing) choked on the marker.
The prefix object and directory key object paths are unchanged: a real
"dir/" object still lists as its own key.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume tier: accept deletes on a volume as soon as it is tiered
volume.tier.upload marks the volume read-only without deletes before the
upload (markVolumeReplicasWritable(..., false, false), i.e. noWriteOrDelete),
and nothing clears that until the volume server restarts, when load() gives
every tiered volume noWriteCanDelete. Every delete in between is refused with
"volume N is read only": the needle stays live in the index, and its bytes in
the remote object, until someone notices.
After the remote file is loaded, the upload handler now gives the volume the
flags a restart would (MarkTieredDeletable: writes refused, deletes accepted),
with two exceptions that keep the current behaviour:
- a replicated volume: the shell copies the uploaded replica's .idx and .vif
to the other replicas after the upload, and a tombstone written before that
copy would reach one replica only;
- a volume in I/O quarantine: that flag is not the upload's to lift.
Tests: TestTierUploadAcceptsDeletesOnASingleCopyVolume fails without the
change ("volume 1 is read only") and checks writes stay refused;
TestTierUploadKeepsRefusingDeletesOnAReplicatedVolume; TestMarkTieredDeletable
(quarantine kept).
* volume tier: check the I/O quarantine under noWriteLock
MarkTieredDeletable read the quarantine state before taking noWriteLock.
CollectHeartbeat could quarantine the volume and set noWriteOrDelete in
between, and the upload then cleared that refusal, so a quarantined volume
accepted deletes until the next heartbeat. Take noWriteLock first: the
heartbeat marks the quarantine before it takes noWriteLock, so either the
check sees it or the heartbeat's refusal lands afterwards and stays.
Also leave alone a volume past IoErrorTolerance that the next heartbeat
will quarantine, as the heartbeat itself does.
* volume: read noWriteOrDelete under noWriteLock in DeleteVolumeNeedle
DeleteVolumeNeedle checked v.noWriteOrDelete without the lock that
MarkVolumeReadonly, CollectHeartbeat and now MarkTieredDeletable hold when
they write it, so a delete racing a tier upload was a data race. Copy the
flag under the read lock; the delete path is otherwise unchanged.
* volume tier: mirror the delete acceptance rules in the Rust volume server
The Rust tier upload already promoted a freshly tiered volume to
no-write-can-delete via refresh_remote_write_mode, and did so for every
volume -- including the replicated and I/O-quarantined ones the Go
MarkTieredDeletable deliberately refuses. A live tier-up now goes
through mark_tiered_deletable (replicated volumes and quarantines keep
their upload-time mark until the next load), while load() keeps
promoting every tiered volume unconditionally, as Go does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable
MarkVolumeWritable preserved noWriteCanDelete on a remote-tiered volume
only when it was already set. A tiered volume still holding the
upload-time mark (a replicated volume, or one whose tier upload ran
before MarkTieredDeletable existed) had it clear, so marking writable
published the volume as fully writable while its .dat lives remotely --
writes then failed against a backend that cannot append, after the
master was told the volume is writable again.
load() gives every remote-tiered volume noWriteCanDelete; deriving the
same flag here keeps mark-writable consistent with a restart: "writable"
on a remote volume resolves to can-delete.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable (rust)
set_writable preserved the remote delete-only mode only when the flag was
already set, so a tiered volume holding the upload-time read-only mark
(replicated, or quarantined under mark_tiered_deletable) was published
fully writable with its .dat remote. Derive no_write_can_delete from
has_remote_file() instead, matching load() and the Go MarkVolumeWritable.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume tier: serialize the quarantine check with the flag swap (rust)
A heartbeat refusal landing between should_quarantine() and the store
could be clobbered until the next heartbeat re-asserted it — the same
race the Go side closed by checking under noWriteLock. A
compare-exchange gives the same ordering: the swap fails, and the retry
sees the quarantine mark the heartbeat sets first.
* volume tier: MarkTieredDeletable refuses a non-tiered volume
A stray call on a local single-copy volume would pin it
read-only-can-delete. The only caller promotes right after
LoadRemoteFile, but the guard makes the method safe to reuse; the Rust
mark_tiered_deletable gets the same check, though its caller already
only reaches it under has_remote.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables, iceberg: log client refusals at V(1), not ERROR
Requests the S3 Tables and Iceberg REST APIs correctly refuse with a 4xx
(access denied, not authorized, no such namespace, already exists, not
empty, a commit conflict) were logged with glog.Errorf, so a client probing
for what it may do, or retrying an optimistic commit, filled the ERROR log
and hid real faults from anyone alerting on it.
- s3tables: HandleRequest wraps the ResponseWriter in a statusRecorder (with
Unwrap for http.ResponseController) and logs a handler error by the status
it answered: 4xx at V(1) as "<op> refused (<status>)", 5xx or no status at
ERROR as before.
- s3tables: the "handleCreateNamespace called" trace moves from ERROR to V(3).
- iceberg: writeManagerError's mapping is split into classifyManagerError,
shared with a new logManagerError that logs 4xx-mapped errors at V(1) and
everything else at ERROR; used by CreateNamespace, CommitTable and
CommitTransaction.
- storage: loading a remote-tier volume logged
"readSuperBlock remote volume N: <nil>" at ERROR even when the read
succeeded; it is now logged only when there is an error.
Tests: TestRequestRefusedWith4xxIsNotLoggedAsError, TestLogRequestErrorByStatus,
TestStatusRecorderKeepsTheFirstStatus, TestStatusRecorderUnwrapsForResponseController,
TestNoTraceLoggedAtErrorLevel (fails on master on the CreateNamespace trace),
TestLogManagerErrorRefusalsAreNotErrors, TestRemoteSuperBlockLogOnlyOnError.
* s3tables: do not log a malformed CreateNamespace body at ERROR
handleCreateNamespace logged a body it could not decode at ERROR before
answering 400; the dispatcher already logs that refusal once at V(1).
Also drop statusRecorder.Flush: it made the recorder an http.Flusher even
over a writer that cannot flush, so http.NewResponseController reported
success instead of ErrNotSupported. Unwrap already lets the controller
reach the wrapped writer.
* iceberg: require exactly one log line per classified manager error
The test accepted a refusal that produced no log, or both logs. Require
one V(1) refusal and no ERROR for a 4xx, and the reverse for a fault.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:00:33 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: regression test for UploadPartCopy ETag on ranged copies
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: hash the copied bytes for a partial-chunk UploadPartCopy ETag
A ranged UploadPartCopy that lands inside a source chunk stored the source chunk's ETag on the destination chunk, so the part reported the MD5 of the wrong bytes. streamCopyChunkRange now tees the forwarded plaintext into an optional digest writer; copySingleChunkForRange feeds it an md5 only when the range is not the whole source chunk, since a whole-chunk copy's stored bytes (and ETag) are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: stamp the copied range's MD5 on UploadPartCopy part entries
The part entry got no Attributes.Md5, so copyEntryETag fell back to filer.ETagChunks: a single-chunk part inherited the source chunk's MD5 and a multi-chunk part reported a <md5>-N composite of source chunks. copyChunksForRange now returns the MD5 of the copied bytes — a single chunk's own digest, or for multi-chunk parts a second ordered stream of the source range hashed beside the concurrent chunk copies (md5 cannot combine per-chunk digests). The handler stores it on Attributes.Md5 and mirrors putToFiler's ExtETagKey, so CopyPartResult, ListParts and the completed object's composite ETag use each part's own MD5 (#11855).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: drop the source chunk's gzip flag on partial-range copies
A ranged UploadPartCopy fetches raw (decompressed) bytes from the source volume and stores them uncompressed, but the destination chunk kept the source's IsCompressed flag, so reads of a partial copy of a gzip-stored chunk would try to decompress raw bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: hash the chunk range, not the entry, for the part MD5
getEncryptedStreamFromVolumes prefers inline Content over chunks, but copyChunksForRange copies the chunks — on an entry carrying both, the hash would describe different bytes than the copy.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: cover sparse ranges in the part MD5 and fail on hash errors
A single chunk's digest is the part MD5 only when it supplies the whole range; a sparse source zero-fills the rest, so gaps are hashed via the ordered range stream, and a fully chunkless range hashes zeros locally. A failed hash pass now fails the part copy instead of storing a chunk-composite ETag that describes the wrong bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: rank copied-range destination chunks by source timestamps
The concurrent chunk copies stamped each destination chunk with an arbitrary finish time, but reads resolve overlapping chunks by ModifiedTsNs — a copy of overlapping source chunks could expose different bytes than the source (and than the part MD5 just hashed). Destination chunks now get timestamps ranked by the source's, preserving its overlap precedence.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: cancel the part-MD5 stream when a chunk copy fails
The hashing goroutine ran on context.Background(), so an early chunk-copy error return left it streaming and hashing the whole source range for a discarded result. A cancellable context now aborts it on every exit path.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: hash the copied range when the covering chunk has no usable ETag
TUS and mount writes leave chunks without an ETag, so a whole-range
UploadPartCopy would otherwise save the part with an empty Md5 and no
ExtETagKey, and multipart completion rejects the part.
* s3api: build the md5 source entry without copying the proto message
* s3api: drop the issue reference from the part-copy etag test
* s3api: route inline-content sources through the plaintext part copy
The raw-byte path walks entry.GetChunks() only, so a ranged
UploadPartCopy of an inline-Content source (filer HTTP or mount writes)
produced a zero-filled part. Send those through
copyObjectPartViaReencryption, which streams the real bytes and stamps
their MD5 on the part.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: store the empty-part MD5 on writeEmptyCopyPart entries
The slow path's empty part returned the right CopyPartResult ETag but
saved no Attributes.Md5 or ExtETagKey, so ListParts and the completed
object's composite ETag still fell back to a chunk-derived value.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: apply Range to HeadObject
* s3: apply Range to a HEAD of a stored directory key
Directory-key entries answer through serveDirectoryContent before
HeadObjectHandler's range branch, so a ranged HEAD returned 200 and the
full length.
* s3: keep the missing-directory check for plain directories only
A stored directory-key marker is an object, so a range on its empty body
answers 416 like on any other zero-byte object.
* s3: let partNumber keep its priority over Range on a directory key HEAD
* s3: serve ranged GETs of a stored directory key
The same branch now applies the validated range to the inline content on
GET, so a directory-key object answers 206 with the selected bytes like
any other object.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: route lookups to the leader master only (#11887)
After a leader change, the volume server kept sending volume and EC
shard lookups to masters that were not the leader:
* currentMaster was set as soon as a heartbeat stream opened, before
the peer said whether it was leader, and kept a follower through the
redirect and its 3s sleep. A follower answers LookupVolume from its
own vid cache, which right after an election can be empty, so reads
went 404 rather than failing over.
* EC lookups used Store.MasterAddress, a second, unlocked copy that the
heartbeat loop set before each dial and cleared on errors, so EC and
plain reads could disagree on the master or have none.
* the receive goroutine called stream.Send itself when the volume
options changed, racing the send loop, and exited silently on
failure, after which leader redirects were no longer read.
Publish currentMaster only once a master is known to be leader: the
peer answered a new stream without naming a leader, which the leader
always does, or a follower named the leader. Compare redirects against
the connected peer. Replace Store.MasterAddress with MasterFn, backed
by getCurrentMaster. Have the receiver ask the send loop for the extra
heartbeat instead of sending it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: ask the masters for the leader after losing the heartbeat
When the heartbeat stream to the leader failed, currentMaster kept the
lost master until a new stream answered, so lookups on a cache miss
kept dialing it through the retry delay even with a new leader already
elected.
On a regular heartbeat error, ask the configured masters for the raft
leader and publish it, within the retry delay rather than on top of it.
Without an answer the last known leader stays: clearing it would only
trade a dead address for an empty one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: stop the heartbeat on a failed assertion
The leader test closed stopChan only once every assertion had passed,
so a failure left the stream open and the fake master's GracefulStop
waited for the whole test run's timeout.
Start the heartbeat through a helper whose cleanup stops it, safe to
call twice, and waits for it to exit. It runs before the cleanups
registered earlier, so the stream is closed before the store and the
fake master go away. Both heartbeat tests use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: clear the leader when no master names one
When the heartbeat lost its master and no configured master named a
leader, refreshLeaderFromPeers kept the lost one. That master is dead
or no longer leader, and a follower answers LookupVolume from its own
cache instead of refusing, so lookups could return locations no leader
vouched for.
Clear currentMaster instead. Lookups then fail fast ("missing address")
until a leader is known again, rather than reading a follower's cache
or waiting on a dial to a dead master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep volumeSizeLimit 64-bit aligned on 32-bit builds
MasterFn (a func, 4 bytes on 386) replaced MasterAddress (a string, 8
bytes) ahead of volumeSizeLimit and moved it to a 4-byte-aligned
offset, where atomic.LoadUint64 panics with an unaligned 64-bit
operation. The atomic.Uint64 type carries the alignment itself.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: publish the answering leader before a duplicate-uuid retry
Read the leader field first on each heartbeat reply. A follower reply
carries nothing else, so it no longer runs through the option updates;
and a leader that rejects the registration over duplicate uuids still
answers lookups, so it stays the lookup target through the retry wait
instead of leaving the previous master in place.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: publish the master only once it is known leader (rust)
* volume: ask the masters for the leader after losing the heartbeat (rust)
* volume: read the leader field first on each heartbeat reply (rust)
* volume: drop the seed fallback from ec shard lookups (rust)
cached_lookup_ec_shard_locations fell back to the static first seed when
no leader was published, so a mid-election EC lookup could land on a
follower and read its stale shard map. Error out like Go's
masterAddress() returning "" instead, via a current_master() getter
matching getCurrentMaster.
* volume: send volume lookups to the confirmed leader, not the seed (rust)
Every HTTP lookup path — proxy/redirect targets, replicated writes and
deletes, batch chunk deletes, remote chunk reads — dialed the static
first seed, so after a failover they kept asking a follower whose vid
map can be stale. Go routes all of these through GetMaster; switch the
lookup calls and their leader-known gates to current_master().
* volume: mark volumes readonly on the leader, retrying the named one (rust)
notify_master_volume_readonly fell back to the seed when no leader was
published, which the peer answers with "Not current leader". It now
uses only the confirmed leader, skips the call in masterless mode, and
on NotLeader asks the seed masters who leads and retries there — the
same path Go's notifyMasterVolumeReadonly takes. The peer leader query
is shared with refresh_leader_from_peers as lookup_leader_from_peers.
* volume: ask the confirmed leader for copy prealloc options (rust)
VolumeCopy's preallocate query went to the static seed; Go issues it
through GetMaster. Route it through current_master() too — when no
leader is known the lookup fails fast and the copy proceeds without
preallocating, as before.
* volume: keep the announced master port forms, not just http (rust)
publish_current_master stripped to http form, so a leader configured
with an explicit gRPC port (host:9333.19999) was redialed on the
default offset. Store the announced host:port[.grpcPort] form like Go's
currentMaster; callers pick to_http_address / to_grpc_address /
parse_grpc_address per use, and Ping admission compares on the http
form.
* volume: fail replicated writes and deletes when no leader is known (rust)
Gating replication on a known leader meant a write could be
acknowledged on fewer copies than required, and a delete could leave
the data readable on the other replicas. Go fails the request when the
replica-location lookup cannot run; drop the gate so the empty-leader
lookup errors the same way.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 13:36:05 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: judge an absent tag target by its canonical ARN
authorizeAbsentTarget matched the bucket policy's Deny statements
against the request's raw resource ARN. The ARN parsers accept any
account and region but resolve the target by name, so a caller with an
identity grant could reformat the ARN, miss the Deny match, and learn
whether a denied name exists — the oracle the absent-target check was
meant to close. Rebuild the ARN from the resolved names and the real
bucket owner and evaluate the Deny against that, matching what the
existing-target path checks.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: cover a foreign-account ARN on an absent denied table
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: keep the absent-target deny check behind a supplied ARN
Callers that pass no resource ARN (ListTables, ListViews,
CreateNamespace) must reach CheckTableOrBucketPermission's admin
shortcut unchanged; running the bucket-policy Deny match first would
deny the admin lists those callers take.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3): set Access-Control-Allow-Credentials on CORS responses (fixes#11885)
* fix(s3): expand wildcard exposed headers for credentialed CORS responses
* fix(s3): keep wildcard in exposed headers for non-credentialed CORS requests
* fix(s3): expose x-amz-tagging-count on credentialed CORS responses
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 13:10:49 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: authorize before revealing a list or tag target exists
ListTables, ListViews and the tag operations read the target's catalog
metadata before any permission check, so a denied caller could tell an
existing name (403 or the resource's own error) from an absent one (an
empty list, or 404). Evaluate the operation's permission with whatever
bucket context could be read and answer a denied caller the same way for
absent, damaged and denied targets.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: answer bucket reads uniformly when the entry is not a table bucket
A plain S3 bucket directory answers GetTableBucket and ListNamespaces with
500 attribute-not-found while a missing name gets 404, and a denied
CreateNamespace is 403 while a missing bucket is 404. Run the permission
check before the absence answer and read not-a-table-bucket as
NoSuchTableBucket.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: cover absent-target answers matching denied ones
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: take the implicit user name from the verified identity
For self-service actions sent without UserName, the implicit user was
resolved by re-parsing the raw Authorization header for a SigV4
credential and looking that access key up. The header is not what
authentication verified: a request can be authenticated by one
mechanism while carrying a header that names another user's access
key, and the action then ran against that other user.
The auth middleware already records the verified identity in the
request context, so use it, on both the standalone iam server and the
embedded IAM surface. When no verified identity is present the name is
left unset rather than guessed.
* iam: test implicit user name resolution
Cover a request authenticated by presigned credentials that also
carries a junk SigV4 header naming another user: the self-service
actions must run as the verified signer. Update the unit tests for the
context-based lookup.
* iam: resolve the implicit user for standalone GetUser too
The embedded surface fills UserName for GetUser; the standalone server
skipped it, so an implicit GetUser on `weed iam` answered NoSuchEntity.
Same verified-identity source as the other self-service actions.
* iam: cover all implicit-username actions in the regression test
Table and partition statistics are Puffin files that the table metadata lists
in its statistics entries; orphan cleanup keeps them (#11651), and its test
places one at metadata/stats-1.puffin. The table bucket layout rule accepted
only *.stats among them, so an engine that names a statistics file *.puffin
had the upload refused with 403 and could not write statistics at all.
The metadata patterns now admit a *.puffin file directly under metadata/,
with the same safe-character rule as the other catch-all patterns. Nothing
else widens: a Puffin file in a subdirectory of metadata/, with a trailing
extension, or outside <namespace>/<table>/ is still refused.
Test: the new metadata/*.puffin cases in
TestIcebergLayoutValidator_AcceptsRealWorldManifestNames and
TestTableBucketFileValidator_PuffinStatisticsOnlyInTableMetadata fail on
master ("invalid metadata file format") and pass with the change; the
rejected cases pass on both.
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* filer.remote.sync: do not upload an entry whose local copy was dropped
remote.uncache and the filer's eviction under storage pressure drop a
synced entry's chunks and clear LastLocalSyncTsNs; remote.meta.sync does
the same when it pulls a newer remote version. When the remote mtime
falls in the entry's mtime second, which is the common case for a small
file synced promptly, shouldSendToRemote falls back to the cleared stamp
and reads the entry as dirty. The dropped chunks make the update a
content change, so the chunkless entry was uploaded and a zero-byte
object replaced the remote one, the entry's only content.
Skip an update that leaves the entry at the same key with no local data
and a sized RemoteEntry: the object at that key already is the content.
The stamp stays cleared, so remote.cache and eviction still see the
entry as uncached. Renames keep going through completeRemoteOnlyRename,
metadata-only edits of a remote-only entry still reach the remote, and a
rewrite in the remote mtime's second holds local data and is uploaded.
* filer.remote.sync: upload a truncate that leaves the entry remote-only
A FUSE truncate of a cached, remote-backed file to zero (truncateEntry,
or SetAttr with size 0) drops the chunks and keeps the sized
RemoteEntry, so the entry reads as remote-only and the dropped-copy skip
left the remote object with its old content.
Skip only while the entry's size is still the remote object's. Uncache
and eviction keep the size and remote.meta.sync sets it from the
object, so those stay skipped; a truncate changes the size and is
uploaded, in the remote mtime's second too.
- Go/Rust: run the unindexed .dat tail repair on noWriteCanDelete volumes
too — deletes still append tombstone records, so the same crash window
applies and they previously degraded to fully read-only.
- Go readTailNeedle: reject a positive size of 1-3 for v2+ as corrupted
before ReadBytes slices the body's 4-byte data-size field and panics;
a torn last record is now cut instead of crashing the load.
- Tests: TestVolumeLoadRepairsTailOnCanDeleteVolume and
TestVolumeLoadCutsDamagedSmallSizeTailOnCanDeleteVolume (Go), and the
Rust mirrors volume_load_repairs_tail_on_can_delete_volume /
volume_load_cuts_damaged_small_size_tail.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: repair an unindexed .dat tail at load
Mirror Go's repairUnindexedDatTail (#11870): a crash between the .dat\nappend and the .idx append leaves a .dat tail no index entry covers,\nwhich sent the volume read-only on the next load and left acknowledged\nneedles unreadable. Before the index loads, re-index the complete\nneedles in the tail (a size 0 record as the tombstone or empty write\nresolveEmptyTailRecords reads it as) and cut only an incomplete last\nrecord. A tail an interrupted append does not explain is left as it is\nand the integrity check quarantines the volume read-only as before.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: tests for the unindexed .dat tail repair
Reload volumes through the real load path after dropping .idx entries\nand cutting or damaging the .dat, mirroring Go's\nvolume_checking_tail_repair_test.go: complete unindexed needles are\nre-indexed (deletes as tombstones, empty writes as empty needles), an\nincomplete last record is cut, and a tail an interrupted append does\nnot explain is left untouched with the volume read-only.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: re-index a torn size 0 record as a live empty needle
A delete and a write with no data append the identical size 0 record, so a
recovered one cannot be told apart from the prior index state alone: the
write path accepts an empty overwrite of a live key and indexes it with
size 0, while a delete of the same key publishes a tombstone. Inferring a
tombstone from prior live state turned a lost index row after an empty
overwrite into a deletion -- the object read back not-found although the
write was answered.
Every re-indexed record is now indexed at its on-disk size, the entry
publishWriteRequest writes for an empty write. The other misread is
bounded: a recovered delete reads its key back as an empty object rather
than not-found, and the size 0 record carries no data to resurrect.
Review feedback on the Rust mirror (#11878): the same bug exists here;
fixing both sides identically.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: re-index a torn size 0 record as a live empty needle
Mirrors the Go fix in this pull request. A delete and a write with no data
append the identical size 0 record, so resolve_empty_tail_records could
not tell them apart from the prior index state: it treated a zero-size
record of a live id as a tombstone, although the write path indexes the
same bytes as a live size 0 entry. A crash after an empty overwrite of a
live key therefore recovered the object as deleted.
Every re-indexed record is now indexed at its on-disk size, the entry
publish_write writes for an empty write. A recovered delete reads its key
back as an empty object rather than not-found; the size 0 record carries
no data to resurrect.
Review: greptile-apps[bot] and devin-ai-integration[bot] on #11878.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: repair a .dat tail left past an empty .idx
The writable index loader creates the .idx before the first needle append,
so a crash after that append leaves a volume whose data nothing indexes.
CheckVolumeDataIntegrity skipped an empty index outright, so the volume
loaded writable and the surviving first needle stayed invisible while
later writes appended past it.
A zero-size index now runs the same repair with tailEntryPos -1: the walk
starts at the first record past the superblock with no anchor needle and
no append timestamp to compare against, gets the same complete-record
validation and incomplete-tail policy, and re-verifies through the normal
tail check once it wrote entries. A tail the repair cannot explain fails
the check, so the volume loads read-only instead of writable on an
unaccounted-for .dat.
Review feedback on the Rust mirror (#11878): the same bug exists here;
fixing both sides identically.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: repair a .dat tail left past an empty .idx
Mirrors the Go fix in this pull request. The writable index loader creates
the .idx before the first needle append, so a crash after that append
leaves a volume whose data nothing indexes. Both the repair and the
integrity check skipped an empty index, so the volume loaded writable and
the surviving first needle stayed invisible while later writes appended
past it.
An empty .idx now runs the same repair with no anchor: the walk starts at
the first record past the superblock, and a tail it cannot explain fails
the load like any other repair failure, since every later check skips an
empty index.
Review: devin-ai-integration[bot] on #11878.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: sync the .dat before publishing re-indexed .idx rows
A power loss after the repaired .idx rows become durable but before the
.dat tail records they name are flushed leaves the index pointing past
the end of the data file; the next load then keeps the volume read-only
until an operator steps in. Sync the .dat first so the rows never
outlive the records they reference.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: sync the .dat before publishing re-indexed .idx rows
Mirrors the Go fix: a power loss after the repaired .idx rows become
durable but before the .dat tail records they name are flushed leaves
the index pointing past the end of the data file. Sync the .dat first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 23:13:51 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: compare parquet schemas by field id in bin-pack merge
parquet.EqualNodes ignores Iceberg field ids, so a column dropped and\nre-added under the same name and type compared equal and was merged,\nfiling the new column's values under the dropped field id. Require the\nfield ids to agree wherever both sides carry one; the v3 layout's\nper-field check uses the same test.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: merge bin files whose schemas differ by field id
A bin used to take the first file's Parquet schema as the writer's and\nrefuse every file that was not structurally identical, so a table never\ncompacted once files had a different column set or REQUIRED-vs-OPTIONAL\nrepetition. A file whose flat columns the writer's schema covers by\nfield id is converted into it (a column the file lacks reads as null);\na file that covers the writer's schema restarts the bin with it, so no\ncolumn is dropped. Equality-delete columns resolve on each file's own\nschema. Renames, type changes, nested columns and files without field\nids stay refused. The sorted and spatial merges are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: test bin-pack merge across parquet schema spellings
Regression coverage for the field-id merge: an added column in either\norder, REQUIRED vs OPTIONAL, equality deletes resolved per file, the\nre-added column refused on v1/v2 and v3, and the refusals that stay\n(rename, type change, one-sided columns, an added required column, no\nfield ids).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: resolve missing delete columns as null in bin-pack merge
An equality delete on an optional column added after a data file was\nwritten no longer refuses the bin: the file reads the column as null,\nso a real delete key cannot match its rows and a null key deletes them.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: write bin-pack merges under the files' union schema
The writer schema is now the union of the bin's file schemas by field\nid, computed in a schema-only pass before any row is written: a column\nstays REQUIRED only where every file has it and spells it so, and the\nmerge refuses only true name/type/id conflicts. Files that need a union\nneither side covers (an added column plus a REQUIRED/OPTIONAL\ndifference, disjoint added columns) now merge in a single pass instead\nof refusing, and progressively wider bins no longer reread and rewrite\nevery earlier file per restart.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: spell the re-added column's test file for codespell
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: resolve equality-delete columns by field id
A column renamed after its data and delete files were written still carries
its field id there; resolving it by id instead of the current name keeps a
rename from turning every equality key null. The missing-column null-key
fallback is now for data files only: an equality-delete file that lacks its
equality column is refused, as the spec requires it to carry them.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: read only the parquet footer choosing a bin's writer schema
The union-schema pass downloaded every input in full, then the merge
downloaded each again. The footer suffices for the schema, and a chunked
file's ReadAt pulls only the chunks the footer spans.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer_pb: add MkdirExclusive
A taken name leaves the callback as a result, not an error, so filer
failover does not count a lost race as a filer failure.
* s3tables: create the entry with its attributes in one exclusive create
CreateTableBucket and CreateNamespace checked that the name was free and
then wrote the entry in several calls, so a second create in the window
replaced the first caller's entry including its owner. The create now
carries the marker, metadata and tags in one O_EXCL CreateEntry that
answers ErrEntryAlreadyExists with the same 409.
* s3: PutBucket uses an exclusive mkdir
The same window let a plain PutBucket replace a table bucket created
concurrently. The already-exists branch now answers from the entry itself,
since the registry just cached the name as absent, and the admin
auto-create path is exclusive too.
* s3: read the winning entry when a create loses the race
The concurrent-create branch called existingBucketError on a name the
registry had just cached as absent, answering NoSuchBucket, and the
auto-create fallback could assign an S3 owner to a table bucket it met
there. The negative cache is cleared before the existing-bucket error is
built, and a table bucket keeps its owner-less shape.
* s3: fail an auto-create that met a table bucket
A table bucket carries no S3 owner and must not gain one, and the upload
that auto-create ran for must not continue into it either.
* s3: fail an auto-create when the winning entry cannot be read
An unreadable existing entry fell through to success, so the write
continued into an unvalidated entry, possibly a table bucket.
* s3: resolve versionId tagging requests to the version-tagging actions
A request for an object's tags with versionId acted on a historical
version but authorized against the current-object tagging action, so an
explicit Deny on s3:*ObjectVersionTagging never applied.
* iam: map PutObjectVersionTagging and DeleteObjectVersionTagging to the tagging action
* s3: only resolve tagging when the query value is empty
The tagging routes match ?tagging with an empty value, so a non-empty one
is not a tagging request and would resolve a tagging grant onto the plain
read or version read that actually serves it.
* s3: keep resolving tagging for any query value
The routes match ?tagging whenever the key is present, so tagging=x is a
tagging request served by the tagging handlers and must resolve to the
tagging actions.
* s3tables: only the admin repairs a table bucket that has no owner metadata
CreateNamespace wrote bucket metadata naming the caller as owner before any
permission check ran, so an ordinary caller that found the marker without
metadata was written in as the bucket's owner and passed every check after.
A bucket in that state is answered with the operation's own denial; the
admin or a trusted internal handler may still repair it.
* s3tables: repair the marker-only bucket under the entry lock
A creator finishing its own create between the missing-metadata check and
the repair write would have its owner overwritten. The mutation now adopts
whatever metadata is there and writes only while it is still absent.
* s3tables: gate the repair on the Admin action, not an operation grant
hasAdminAction also answers s3tables:*, so a non-admin identity holding
the wildcard operation grant could pass the repair check and write itself
in as the bucket's owner.
* s3api: persist single-identity IAM changes through the credential manager
CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy, DeleteUserPolicy, SetUserStatus, UpdateUser and DeleteUser used to load the whole configuration, change one identity and then SaveConfiguration, rewriting and re-propagating every identity. They now write only the identity (and, for rename/delete, the groups and service accounts that referenced it) through the credential manager, which propagates that identity alone. Static identities and skipPersist keep the previous full-save/no-write contract.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iamapi: persist single-identity IAM changes through the credential manager
Same targeted-write change as the embedded IAM API for the standalone weed iam dispatcher: UpdateUser, DeleteUser, CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy and DeleteUserPolicy write one identity through the credential manager instead of saving the whole configuration. CreateAccessKey keeps the full save when the handler created a user the store does not know.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* credential/memory: re-key identity on UpdateUser rename
UpdateUser stored the identity under the old username key even when identity.Name changed. Re-key it the way filer_etc does (delete old, save new) so the targeted rename path sees the moved record.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover targeted single-identity IAM writes
Recording-store tests assert each rerouted action uses the single-identity credential store methods and never SaveConfiguration, in both the embedded S3 IAM dispatcher and the standalone iamapi dispatcher, plus the skipPersist and static-identity contracts and a memory-store rename case.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* credential: keep not-found and rename-collision semantics in stores
IamGrpcStore.DeleteAccessKey translates gRPC NotFound to ErrAccessKeyNotFound so a repeat delete stays the no-op the IAM handlers advertise. MemoryStore.UpdateUser rejects a rename onto an existing user instead of overwriting it, matching RenameUser.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 17:41:37 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: repair an unindexed .dat tail at load instead of loading read-only
A needle write appends the needle to the .dat, then its entry to the .idx,
then answers the client. A crash between the two appends leaves a .dat that
ends past the last indexed needle; on the next start CheckVolumeDataIntegrity
reports "actual N bytes expected M bytes" and the volume loads read-only, and
a complete needle in that tail cannot be read because nothing indexes it.
After a process crash those bytes belong to a write that was never answered;
after a power loss a complete needle can survive in the .dat while its .idx
entry did not, although the client was answered. So complete needles are
re-indexed rather than cut.
CheckVolumeDataIntegrity now repairs the tail before the existing check,
which then runs on the repaired files as before. The repair:
- acts only on a local, version 3 volume within MaxPossibleVolumeSize, with a
whole number of .idx entries, that is about to load writable (neither
noWriteOrDelete nor noWriteCanDelete is set);
- requires the needle at the highest indexed offset to match its entry (id,
size, whole record, CRC);
- walks from the end of that needle to the real end of the file and gives
every complete needle (header, body, checksum, timestamp and padding inside
the file, CRC valid, append timestamp later than the needle before it) the
.idx entry the volume server would have written, in .dat order; a size 0
record, which is how a delete is appended, is indexed as a tombstone;
- cuts only an incomplete last record: fewer bytes than a header, a record
running past the end of the file, or a last record whose CRC fails;
- changes nothing otherwise (a damaged record with more data after it, a
header with id 0 or a negative size, append timestamps going backwards), so
the volume loads read-only exactly as before.
The .idx entries are written and synced before the .dat is truncated and
synced, so a crash during the repair leaves the needles indexed and the
remainder still past them; the next load cuts it. One warning per repaired
volume names the needles re-indexed and the bytes cut.
Tests: volume_checking_tail_repair_test.go reloads volumes through the normal
load path. Repaired: one and two complete unindexed needles, an unindexed
delete, a partial header, a partial body, a complete needle followed by a
partial one, a last needle with a bad CRC, a partial needle after an indexed
tombstone; each reloads clean afterwards. Left read-only and unchanged: the
last indexed needle with a wrong id or a bad CRC, a bad CRC followed by more
data, zero bytes past the last needle, an index past the data, a volume
persisted read-only, a partial .idx entry, and a clean volume.
TestCheckVolumeDataIntegrityVerifiesDeletionTail now also checks that on a
writable volume its 64 zero bytes are reported and kept.
Fixes#11864
* volume: fail the integrity check when a tail repair cannot complete
A repair that wrote the .idx or cut the .dat and then failed to sync only
logged the error; the check that followed saw the repaired files and the
volume loaded writable on a repair that may not be on disk. Any repair
error other than an unexplained tail now fails the check, so the volume
loads read-only, and the next load repairs again.
A failed .idx append could leave part of an entry behind, which made every
later load reject the .idx as a partial entry. The appended bytes are
truncated away again before the error is returned.
* volume: do not cut a tail record that could not be read
The walk cut a last record whose read failed for any reason, so an I/O
error (EIO) on a complete record erased it. Only a record whose bytes are
damaged (needle.ErrorCorrupted: a CRC mismatch or a body that does not
parse) is cut now; any other read error is returned with both files left
as they are, and the volume loads read-only.
* volume: treat a sized tail record with no data as damaged
A needle with no data is written with size 0. A tail record whose header
gives a body but whose data size field reads 0 parsed as an empty needle,
and ReadBytes checks the CRC only when there is data, so the repair indexed
it with its positive size and reads returned an empty body. Such a record
is now damaged like one with a bad CRC: cut when it is the last record,
otherwise the tail is left alone and the volume loads read-only.
* volume: keep an unindexed empty write as an empty needle
A delete and a write with no data both append a size 0 record, and the
repair indexed every size 0 record in the tail as a tombstone, so an
acknowledged empty write whose .idx entry was lost read back as deleted.
A delete is only appended for a needle that is live, so a size 0 record
of a needle that is absent or deleted (in the .idx and in the tail before
it) is an empty write and is re-indexed with size 0. One of a live needle
stays a tombstone, the reading weed fix and the backup's index
regeneration already give it. The .idx is scanned only when the tail holds
a size 0 record.
* volume: cut torn-tail test records inside their body, not at +40
The partial-record cases cut the .dat 40 bytes into a record. A needle
with up to 7 bytes of data is a 40-byte record, so newRandomNeedle's
random length (1..1023) left it whole about once in 150 runs per case:
the repair then correctly re-indexed it and cut nothing, and the test
failed on the .dat size. The cut now lands halfway through the record's
body, which is always past its header and short of its end.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* volume: run the durable sync outside the reader-visible lock
Reads of a volume waited on that volume's fsync because the write lock
was held across Sync()/sync_all. A new per-volume lock (Go
dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat
mutation and lifecycle op across append+publish+sync+rollback, while the
fsync itself runs with dataFileAccessLock/the io write guard dropped —
so readers proceed during a sync, and a failed sync's rollback still
cannot clobber another write or a compaction swap.
Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk)
and Rust (write_needles_grouped/write_synced_run).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: publish after sync; fix Close/CommitCompact lock order
Address review on #11863:
- Close took dataFileSyncLock before waiting on isCompactionInProgress
while CommitCompact claims the flag then takes the lock — a deadlock.
Close now claims the flag before taking the locks (flag -> sync -> access).
- Go now publishes needle-map entries only after the .dat sync, matching
Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes,
publishWriteRequest/publishDeleteRequest index them after Sync. A failed
sync rolls back bytes no reader ever resolved — the streamed-read
offset-invalidation window is gone. processBatch splits runs at repeated
needle ids (Rust parity) so dedup/cookie checks stay sequential; the old
snapshot/restore rollback machinery is removed.
- Rust commit_compact (store-lock caller) takes data_file_sync_lock via
try_lock -> WouldBlock before the write lease; commit_compact_held and
the tier-down _held paths document/require the caller-held sync lock,
and the two vol.instance().write_lock() grpc sites take it first. The
swap can no longer land between a durable write's append and publish.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: restore write watermarks when an inline fsync fails
The syncWrite rollback truncated the .dat but left lastAppendAtNs,
lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and
lastModifiedTsSeconds describing the rejected write; recovery and
compaction could read a watermark for a record that no longer exists.
Snapshot and restore them on failure, as processBatchRun already does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 12:58:02 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: mask anonymous misses behind AccessDenied unless the bucket is listable
Deferred anonymous GET/HEAD authorization only runs once an entry is
selected, so misses returned 404 while existing keys returned 403 — an
unauthenticated existence oracle for bucket names and object keys.
Route not-found responses in Get/HeadObjectHandler through a check that
only discloses 404 when anonymous listing is allowed (policy, identity or
a public-read bucket); otherwise the miss is indistinguishable from a
denied read, matching AWS and 4.48 behavior.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: mask missing-directory misses for anonymous reads too
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: check a fixed request shape when masking anonymous misses
The miss check passed the caller's request through, so query parameters
could retarget the resolved action (?policy) or satisfy an s3:prefix
condition on an unrelated subtree. Each check now runs on a request with a
fixed shape: a ListBucket scoped to the probed key and a plain read of the
key, so the miss only shows when the caller could learn the same fact
directly.
* s3: mask conditional-header misses for anonymous reads
An If-Match conditional on a missing key wrote NoSuchKey directly from
processConditionalHeaders, before the handlers reached the masked miss
path, so anonymous callers could still distinguish a missing key from a
denied read.
* s3: keep the versionId when masking anonymous misses
The read leg must evaluate the action the caller's request resolves to:
a versioned miss is s3:GetObjectVersion, and dropping versionId would let a
plain GetObject grant disclose versions the caller may not read.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The auto-select branches of ReceiveFile (is_ec_volume) and
VolumeEcShardsCopy passed the build-default DATA_SHARDS_COUNT into
find_ec_shard_target_location, so its free-slot maths scored every
volume as 10+4 even when the volume's .vif declares a custom ratio
(e.g. 6+2). A wider-ratio volume's shards could be steered onto a disk
too full to hold them all, and a narrower one would be refused slots
it actually has.
Add Store::lookup_ec_data_shards(collection, vid), which answers from
the mounted EC volume first and then from the .vif on whichever disk
holds it (via read_ec_shard_config_across_dirs, so split -dir/-dir.idx
and rebuild-staging layouts are covered), falling back to the default
when no metadata exists yet.
Mirrors the enterprise fix (seaweed-mono c22fa8a15).
* master: detect and quarantine duplicate volume ids across layouts (#11763)
* shell: volume.check.duplicate_ids audit command (#11763)
* master: persist duplicate-vid quarantine across heartbeats (#11763)
* master: quarantine all copies of a duplicated volume id
Near-simultaneous registrations let each side quarantine only the other,
so arrival order decided which copy stayed writable. A confirmed
cross-layout collision means either copy could be the impostor, so flag
every colliding copy read-only regardless of registration order.
* topology: return a copy from VolumeLayout.Lookup
Callers iterating the returned slice raced with registrations and
removals replacing the location list in place.
* master: release duplicate-vid quarantine when the conflict resolves
A cross-node volume.move briefly shows the same id on two nodes under
different disk layouts; permanent quarantine would pin a valid target
read-only. Keep the first-reported state per quarantined copy and lift
the marks once the conflicting copy unregisters, so only a persistent
collision stays read-only.
* shell: include ec shards in volume.check.duplicate_ids
Ec-only volumes share the global id space. Compare shard collections
against regular ones while allowing shards and regular copies of the
same encoded volume.
* master: keep quarantine release current and cover all removal paths
Track the latest server-reported read-only flag while quarantined so a
release restores the server s true state, and run the release check on
collection deletion and data-node removal too, not only explicit volume
unregistration.
* shell: do not flag ec-only volumes as duplicates
An ec-only volume has no regular copy, so its shard collections had
nothing to compare against and always read as foreign.
* master: quarantine every conflicting layout at the layout level (#11763)
- collect all conflicting layouts, not just the first, so a vid spanning
three layouts does not leave one writable
- mark quarantine on the location list only: folding it into the stored
report made the volume digest permanently disagree with the server and
forced a full volume list every heartbeat
- isAllWritable now honors the location read-only mark so a quarantined
copy stays out of writables through resyncs and re-registration
- re-check the conflict before keeping marks, and do not clear a node's
quarantine while it still holds the vid in another layout
- log the full layout keys on both sides of the conflict
* master: push duplicate-vid quarantine to the volume server (#11763)
A quarantined copy stayed writable on the volume server itself for a
client holding an existing file id, since the master-only mark never
reached the server's local write path. Quarantine now also calls
VolumeMarkReadonly (can_delete so expiry still drains), is re-pushed if
the server comes back reporting it writable, and VolumeMarkWritable
runs when the conflict resolves and the server had reported writable.
* master: retry unenforced quarantine marks and restore server writability on release (#11763)
Dial the advertised grpc port, return mark errors, and keep a
quarantine entry until the writable push succeeds so a released
copy does not stay read-only on the server.
* master: treat rp-only layout differences as an in-flight replication change (#11763)
volume.configure.replication rolls through nodes one at a time, so a
volume legitimately reports the old and new replica placement while it
settles. Layouts differing only in rp within the same collection are a
supported transition, not a reused id.
* shell: group ec shards by collection in the duplicate id audit (#11763)
EC shards of one volume legitimately span disk types when
DiskTypePrefer spills, so grouping by disk type reports healthy
volumes as duplicates.
* pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763)
* master: freeze volume growth on topology id mismatch (#11763)
* shell: volume.growth.resume command (#11763)
* master: replicate the growth freeze and gate it on veteran heartbeats (#11763)
* master: block allocation while the topology id is being minted
DeferAllocation only armed after the topology id persisted, so growth
requests arriving during init ran ahead of the warmup window. Gate
NextVolumeId on an init-pending flag spanning the whole mint.
* master: stash reported topology ids until the local id is minted (#11763)
* drain stashed topology id reports when the id arrives outside the mint path
* persist the growth-freeze flag in the mirror file and raft snapshots
* log hashicorp raft marshal and apply errors in applyGrowthFreeze
2026-10-20 07:34:31 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: group-commit fsync uploads on the direct write path
Each fsync=true upload used to run its own .dat/.idx sync when the write
queue was off, which is the default. Route durable uploads through the
per-volume write workers so a batch shares one sync, matching Go's
writeNeedle2: non-fsync writes, and writes while the server is stopping,
stay on the synchronous path. The queue is now always constructed and
SEAWEED_WRITE_QUEUE is gone.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: document post_handler write routing
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: share the UpdateTable authorization gate
Extract the validation, load, and permission check from handleUpdateTable
into authorizeUpdateTable so the table an update was authorized against is
the same snapshot the mutation writes to. Manager gains
AuthorizeUpdateTable, mirroring AuthorizeCreateTable, so callers can ask the
question without updating anything.
* s3tables: share the UpdateView authorization gate
Extract the lookup, policy load, and permission check from handleUpdateView
into authorizeUpdateView, mirroring authorizeUpdateTable, and expose it as
Manager.AuthorizeUpdateView for callers that need the answer without the
update.
* s3api: scope vended table credentials to read-only on request
VendTableCredentials takes a readOnly flag and tablePrefixSessionPolicy
emits only s3:GetObject for the table prefix when it is set, keeping the
conditioned ListBucket and unconditioned GetBucketLocation statements that
reads still need.
* iceberg: vend read-only credentials when the caller may not write
LoadTable and friends vend storage credentials only for the authority the
caller holds: the vend now checks UpdateTable/UpdateView permission and a
caller that may only read the catalog entry gets a read-only session. Paths
that just passed a write gate (stage-create, view commit) keep the full
session.
* lance: vend read-only credentials when the caller may not write
storage_options vends the same read-only session as the Iceberg catalog
when the caller lacks UpdateTable on the table.
* lance: keep write credentials for callers authorized to declare
A Lance commit is a direct S3 write that never revisits the catalog, so a
caller that just passed CreateTable still gets a writable session; the
UpdateTable gate applies to describe, the read path.
* iceberg, lance: log a failed write check instead of only reading it as denied
2026-10-20 07:16:27 -07:00
Chris LuGitHubDevin <devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: list keys in byte order around directory markers
* s3: order deferred dir blocks by slash name and hold quota
* s3: fix deferred dir blocks at page and marker boundaries
Review follow-ups on the marker byte-order change:
- A marker inside an emitted "<name>/" block re-descended and re-listed its
folded contents under delimiter "/" instead of skipping the block.
- A prefix object's "<name>/" common prefix was emitted eagerly at the name
position, outranking siblings like "a.txt" and repeating across pages; the
plain key now emits alone at the name position and the block emits the
common prefix when children exist.
- A recursion starved on quota before listing a deferred subtree returned
without truncation, dropping keys silently; the block now reports a
resumable truncation when the subtree still holds entries.
- Clone prefix-object entries with proto.Clone rather than copying the
protobuf message by value.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: close the hasEntries probe stream on return
* s3: resume delimited listings at the dir block boundary
A deferred directory block emits at its "<name>/" position, but emitDir
left nextMarker at the bare name, which sorts inside the (name, name/)
band and reads as "block pending". Every pass of the listing loop then
re-emitted the same common prefix until maxKeys ran out (999 prefixes,
IsTruncated), and a "<name>/" continuation token trimmed to "<name>" by
adjustMarkerForDelimiter reopened the block and paged forever.
Advance nextMarker to the block boundary the same way the recursive
branch does, and keep a "/" -ending marker intact — it already names the
bound past the block.
---------
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
heartbeatResponse now reads MaxVolumeId and TopologyId off ms.Topo, but
the test built a MasterServer without one and panicked, failing the
package for every PR.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume balance: unify the per-move convergence guard
Evaluate the post-move utilization comparison through serverUtilization so
the guard also applies when servers lack MaxVolumeCount info (raw-count
mode), where it previously did not run at all.
* volume balance: restrict fallback destinations to improving servers
The score-based fallback inside createBalanceTask could redirect a move to
any valid server, including ones as full as the source — between equally
full servers the move only relocates the imbalance and the next detection
cycle reverses it (#11839). Exclude from eligibleTargets any server whose
post-move utilization would exceed the source's, so a fallback destination
satisfies the same convergence guard as the loop-chosen target.
* volume balance: skip a volume when the chosen target violates placement
When the detection loop's target resolves but cannot host the replica (it
already holds one, or is out of the filtered scope), createBalanceTask now
returns nil so the loop advances to the next volume — matching shell
volume.balance, which continues to the next candidate instead of
redirecting the move to a different server.
The score-based fallback is kept for the case where the target cannot be
resolved at all; its allowed set is already limited to servers the move
would improve.
* volume balance: regression test for replica-constrained targets (#11839)
Reproduces the reported loop: the emptiest server holds the second rp=001
replica of the lowest-ID volumes on the fuller servers, so it is never a
legal target for them. Detection must skip those volumes and send the
movable ones to the under-utilized server, and re-running detection after
applying the plan must never move the same volume twice.
2026-10-19 20:52:30 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The master intentionally sheds an assign that triggered volume growth
once its wait window elapses (503 + Retry-After); on a slow CI box
cold-start growth can outlast it, so the single-shot assertion flaked.
Honor the documented retry contract like a real client, within a 60s
deadline.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: strip internal object-lock and version headers on auth entry
putToFiler persists Seaweed-X-Amz-Object-Lock-Mode, -Retention-Until-Date,
-Legal-Hold and -Version-Id request headers into authoritative entry
metadata without a permission check, so an s3:PutObject-only writer could
pin COMPLIANCE retention and legal hold on its own uploads, bypassing the
dedicated-permission gate on the standard x-amz-object-lock-* headers.
The server only sets these headers from validated inputs, so drop them in
authenticateRequestInternal alongside the identity headers already
scrubbed there.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: strip internal SSE headers on auth entry
X-SeaweedFS-SSE-S3-Key-Data, -SSE-S3-Base-IV and -SSE-KMS-Base-IV are read
from the request to drive multipart encryption, so a client could inject
its own key material and IVs. The server sets them itself between handler
stages of a request; drop inbound values in authenticateRequestInternal.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: scrub internal headers before the auth-enabled early return
Auth and AuthPostPolicy skip authenticateRequestInternal when IAM is
disabled, so client-supplied internal headers still reached putToFiler.
Hoist the scrub into scrubInternalHeaders and run it unconditionally in
both wrappers as well as the shared auth entry point.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: test internal header scrub through the auth-disabled wrapper
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 18:35:20 -07:00
Chris LuGitHubDevin <devin-ai-integration[bot]@users.noreply.github.com>
* filer.sync: keep the needle-404 clock from resetting on live volumes
The give-up clock for a source chunk that cannot be produced is kept per
volume so a vacuumed volume's files all inherit one wait. But a needle
404 and a volume lookup failure shared that clock, and sourceServed
cleared it whenever any needle on the same volume read fine, so on a
busy cluster the grace never elapsed for a single deleted needle on a
healthy volume and filer.sync retried it forever (#11841).
Give needle not-found answers their own per-volume clock that serving
siblings does not reset.
* filer.sync: an answered 404 needs no liveness probe before skipping
After the missing-chunk gate gives up, onReplicateChunkError only skips
the entry when a re-read of the most recently served chunk succeeds.
That probe can never have been armed on a fresh sync, and its single
stored file id can itself be vacuumed since — either way the entry stays
pinned and the offset never advances.
A needle 404 was answered by a live volume server, which is itself proof
the source is serving requests, so skip on it directly. Keep the probe
for the lookup-failure flavor, where no server answered and the silence
can still mean a source outage.
* filer.sync: skip a replayed version the source rewrote in place
Supersession was decided by entry mtime alone, so a source entry whose
chunks were swapped without an mtime bump — remote cache eviction and
re-cache, an in-place repair — never read as superseded, and its
vacuumed chunks were retried indefinitely.
When a chunk is missing, compare the replayed chunk's file id against
the live entry's resolved chunk set: absent means the replayed version
is obsolete and its event is skipped on the first failure instead of
after the give-up window. The same live-entry comparison now also
decides the entry-level skip in onCorruptChunk/onReplicateChunkError.
* filer.sync: give each missing needle its own grace clock
A 404 is one needle's answer, but the clock was kept per volume: once the
first needle's wait expired, every other needle on that volume inherited an
already-elapsed clock and could be written off on its first sighting. Track
the wait per file id and clear it only when that needle itself serves again,
so a live volume's healthy needles neither shorten nor restart it.
* filer.sync: keep the probe for filer-proxied 404s
ReadPart proxied through the source filer collapses "no volume locations"
into the same 404 a missing needle returns, so that answer cannot stand in
for the liveness probe the way a direct volume read's can. Only treat 404 as
an answered needle when the sink reads volume servers directly.
* filer.sync: only a not-older live entry can supersede a replay
Chunk-list comparison must respect the same ordering the mtime check does:
a live entry older than the replayed version means the source was rolled
back, so its differing chunks say nothing about whether the replayed data
is still needed. Gate the content check on live mtime >= replayed mtime in
both sourceMovedOn and sourceDroppedChunk.
* filer.sync: compare chunk ids without resolving manifests
resolvedChunkFileIds downloaded manifest files to expand chunk lists, so a
manifest sitting on a dead volume made the supersession check itself fail
and left the offset pinned behind a replica the source cannot read. File
ids are immutable needle identities — the unresolved sets already compare
chunk content exactly, and the check no longer performs I/O.
* filer.sync: judge nested chunks by their top-level chunk
A manifest child is not in the entry's chunk list even when the manifest
still references it, so checking the live entry for the child id reported
it dropped and returned the raw 404 — unwrapped, so the offset still
pinned. Thread the entry-level chunk id down the manifest recursion and
consult it for the dropped-chunk check instead.
* filer.sync: let the manifest-child test exit through the gate
Run the worker with a shortened grace period and assert the returned error
is marked errSourceChunkMissing, instead of racing the worker still running
against a restored RetryWaitTime.
* s3: sort ListMultipartUploads by key and honor key-marker
* s3: skip upload folders without an object key in ListMultipartUploads
* s3: order same-key multipart uploads by initiation time
The upload id suffix is now a time-ordered UUIDv7, so sorting by upload id lists a key's uploads oldest first, like S3. Pagination is unchanged: ordering and the upload-id-marker comparison share the same key.
selectMultipartUploads now builds the whole result, including NextKeyMarker and NextUploadIdMarker, and the pagination test pages with the response markers.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-19 18:10:06 -07:00
Chris LuGitHubDevin <devin-ai-integration[bot]@users.noreply.github.com>
The HEAD on the completed multipart object occasionally gets a 403 from a
signature verification mismatch before the object read settles; retry the
download instead of failing the run.
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
* pb: heartbeat fields for the federated volume-id bound (#11763)
Heartbeat carries max_volume_id_seen and last_seen_topology_id up, and
HeartbeatResponse echoes max_volume_id and topology_id down, so the
allocation bound is replicated to every volume server.
* master: echo the volume-id bound and fold reported bounds in (#11763)
Every heartbeat response carries the leader current max volume id and
topology id; every heartbeat reported max_volume_id_seen is folded into
the topology bound via UpAdjustMaxVolumeId.
* volume: persist a per-dir vid stamp of the highest id ever seen (#11763)
The store keeps max_volume_id_seen + last_seen_topology_id in a
.vid_stamp file per volume dir, fed by heartbeat echoes, AllocateVolume,
and local volume/EC creation, seeded at boot from stamps and local
holdings, and reported in every heartbeat. A rebuilt master recovers
the bound from the first surviving veteran report.
* volume-rust: mirror the vid stamp and heartbeat echo handling (#11763)
Same mechanism as the Go volume server: .vid_stamp per dir, bound folded
in from add_location scans and allocate/mount calls, echoed fields
applied on heartbeat responses, reported on every heartbeat.
* volume: serialize vid stamp writes and fsync the directory (#11763)
* volume-rust: serialize vid stamp writes and fsync the directory (#11763)
* master: echo the allocation bound to connected servers on change
The bound was only sent at connect time or on digest mismatch, so a
healthy stream never learned later increases. If the high-id server then
went offline and the master lost state, survivors restored a stale bound
and the id could be reissued. Resend the heartbeat response whenever the
bound or topology id moved.
* volume: retry failed vid stamp saves on later updates
A transient write failure left the in-memory bound raised but the stamp
stale, and repeat updates at the same bound never retried. Track a dirty
flag so any later update re-attempts the save.
* volume: cover stamp bootstrap for pre-stamp dirs in test
* volume-rust: retry failed vid stamp saves, mirroring Go
* volume-rust: stamp every dir with the combined bound on add_location
A dir whose own stamp sat below the combined bound kept its stale value
after fold; losing the higher-stamped disk then restarted the server
below ids it had already learned. Rewrite all stamps after each location
folds in.
* volume: stamp mounted volume and ec shard ids (#11763)
* volume-rust: stamp mounted volume ids, mirroring Go (#11763)
* volume: retry dirty vid stamp save on unchanged topology id (#11763)
* volume-rust: retry dirty vid stamp save on unchanged topology id (#11763)
* master: record echoed bound from the sent heartbeat response (#11763)
* s3tables: accept .puffin data files for Iceberg v3 deletion vectors
Iceberg v3 writes deletion vectors as Puffin files under a table's data/
directory, so merge-on-read DELETE and UPDATE on v3 tables were refused
with AccessDenied by the table-bucket upload validator.
Fixes#11842
* s3tables: name all accepted data formats in rejection error
The message stopped at .avro even though .lance and now .puffin are
accepted.
* s3tables: cover deletion-vector filenames in data layout test
Uses the names real engines emit: Spark/Iceberg Java
{seq}-{task}-{uuid}-{counter}-deletes.puffin, UUID-suffixed names, and
the same under a partition directory.
* s3api: check the direct peer against the trusted-proxy allowlist
* s3: verify X-Forwarded-Prefix only from trusted proxies
* s3: note that trusted proxies must sanitize forwarded headers
* s3tables: honor a table policy Deny for maintenance operations
authorizeMaintenanceTable evaluated only the bucket policy, so a Deny in the table policy never vetoed PutTableMaintenanceConfiguration, GetTableMaintenanceConfiguration or GetTableMaintenanceJobStatus even though they act on the table ARN. The stored table policy is now checked for a matching Deny first via deniedByPolicy, a veto helper that keeps the admin exemption, and resource and bucket tags are loaded into the policy context so tag-conditioned Denies evaluate. readBucketPolicy is renamed to readPolicy and reused for the table path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: honor the stored policy's Deny on policy management operations
PutTablePolicy, GetTablePolicy and DeleteTablePolicy only evaluated the bucket policy, so a Deny in the table's own policy was ignored and a bucket-level Allow could strip or replace the policy guarding a table. The existing table policy is now loaded and a matching Deny vetoes the call. PutTableBucketPolicy likewise consults the existing bucket policy for a Deny before replacing it. Resource and bucket tags populate the policy context so tag-conditioned Denies evaluate, and policy writes assert the policy authorized against is still in force, returning a conflict when it changed mid-request.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: apply a table policy Deny to tag operations
TagResource, ListTagsForResource and UntagResource evaluated only the bucket policy; a Deny in the table's own policy is now honored when the target ARN is a table. ListTagsForResource and UntagResource also returned early when the resource carried no tags attribute, skipping the permission check entirely; the check now always runs. Policy evaluation uses the canonical resource ARN rebuilt from stored metadata, so a request ARN with a forged account or region cannot dodge the Deny.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: regression tests for resource policy denies
Cover the case where a bucket policy grants an identity s3tables:* while the table policy denies it: the policy, tag and maintenance operations must stay denied so the protection cannot be stripped. Also cover a bucket policy that denies PutTableBucketPolicy under the open fallback, and a tag-conditioned Deny, which vetoes only while the stored tag matches.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: give Volume interior locks mirroring Go's
Volume's mutable state moves behind interior locks matching Go's
per-volume lock domains, so the store can hand out shared handles
instead of requiring exclusive &mut access:
- io: RwLock<VolumeIo> owns the .dat/remote handles, needle map,
append cursor, compaction bookkeeping and write watermarks —
Go's dataFileAccessLock-adjacent mutable set; helpers take the
guard as &mut VolumeIo so nothing re-locks.
- super_block and volume_info each get a RwLock
(superBlockAccessLock, volumeInfoRWLock); dir_idx gets its own
so path building never contends with I/O.
- no_write packs Go's noWriteLock pair into AtomicU8;
last_modified_ts_seconds and compaction_byte_per_second become
atomics.
The data-file write lease (dataFileAccessLock) now covers every
mutating entry that previously relied on borrow exclusivity:
writes, deletes, blob writes, set_writable's flag/index swap,
remote-file updates, local-backend swaps, index relocation,
replica-placement rewrite, put_needle_index, close and destroy.
set_read_only_persist leases only its can_delete idx-attach branch
because mark_io_unavailable reaches it under a held lease.
commit_compact takes try_write_lock — callers hold the store lock,
so blocking would stall every volume behind one slow reader;
WouldBlock surfaces as COMMIT_WAITS_FOR_READS for the caller to
retry.
Lock order is fixed: lease -> io -> volume_info/super_block/dir_idx.
* storage: mount volumes as shared Arc<Volume> handles
DiskLocation's map owns Arc<Volume> so the store can hand a caller a
handle that stays valid across store-guard releases, instead of
borrowing &mut Volume under the store write lock.
- volumes: HashMap<VolumeId, Arc<Volume>>; find_volume returns a
cloned Arc.
- find_volume_mut and volumes_mut are gone — every Volume method is
&self now, so shared handles cover what exclusive borrows did.
- writable_volume_mut -> writable_volume(&self) -> Result<Arc<Volume>>
keeps the same admission checks (exists, disk-space-low).
- write_volume_needle/write_volume_needles/delete_volume_needle are
&self; lifecycle methods that mutate the map or mounted state stay
&mut self.
* server: run needle I/O with the store guard released
The store write lock previously wrapped needle writes and deletes
end-to-end: process_batch held it across every grouped fsync, and the
HTTP/gRPC fallbacks, tail receiver and fetch-and-write paths did the
same. One volume's slow I/O then stalled admission and metadata on
every other volume.
- write_queue resolves each group's handle via writable_volume under
a brief store read, then runs write_needles_grouped after the guard
drops.
- HTTP write/DELETE fallbacks, the gRPC batch-delete EC recheck +
delete, tail receiver and fetch_and_write local writes, blob writes
and the quarantine/sync/set_writable/remote-file paths all run
under store.read() now that every volume method they reach is
&self.
- Lifecycle calls that mutate the volume map or mounted state
(mount/unmount/delete, EC shard ops, compaction commits,
load_new_volumes, heartbeat volume actions) keep store.write().
* test: cross-volume admission proceeds during another volume's write
Regression coverage for the store-lock fix:
- write_queue's test parks a volume-1 write inside its .dat append via
a new needle_write_hook seam (mirroring needle_read_hook), then
proves the store still hands out a lifecycle writer and a volume-2
handle — while volume 1's data-file lease still excludes same-volume
writers.
- tests/store_lock_repro.rs exercises the public API: concurrent
read-guarded writes on two volumes, and a stale Arc<Volume> handle
failing closed after unmount instead of writing into torn-down
files.
* test: drop store write guards that shared handles made read-only
Test setup sites held store.write() purely to reach find_volume_mut;
with Arc<Volume> and &self methods the guard is read-only (clippy
readonly_write_lock). The volume lease still serializes the writes
inside.
* volume: drop a windows-only read_exact_at that shadows the io helper
cargo's windows build failed E0255: the free fn copied with the
interior-lock conversion collided with the unconditional
crate::storage::io::read_exact_at import, which already carries the
platform branches internally.
* server: release the store read guard before needle I/O
The read guard still ran through write_needle_blob_and_index, the tail
receiver write, the fetch-and-write local write, and the HTTP write and
DELETE fallbacks, so a slow append blocked mount/unmount writers (and,
on writer-preferring locks, readers on other volumes). Resolve the
Arc<Volume> under a short guard and write through the handle after it
drops — the same scoping process_batch already used.
* volume: serialize the read-only mark and its .vif save under the lease
set_read_only_persist only leased the idx-attach branch, so concurrent
mark-readonly and mark-writable calls could interleave flag updates and
.vif writes — including two save_vif calls truncating and writing the
file at once. The lease now spans the whole flag change and save, with
a _held variant for mark_io_unavailable, which reaches this from write
paths already holding the lease (Go: markVolumeReadonly runs under
dataFileAccessLock).
* volume,server: span the tier-down transition under one write lease
update_remote_files cleared the remote-only write restriction and
released its lease before open_local_dat_backend attached the local
file, so a write could slip into the gap — pass check_writable, then
fail appending to a missing dat_file. The handler now holds the lease
across the reference trim, .vif save, and backend swap (and the rollback
re-insert), via _held variants; the store read guard is scoped to the
handle lookup. The swap still lands after the .vif trim — the ordering
that keeps a crash from pointing the volume at a deleted remote object.
* server: run a batch's volume groups on separate scoped threads
A single batch can hold writes for several volumes; running the groups
sequentially made every later volume wait behind a stalled append on an
earlier one — the same head-of-line blocking the store guard release
fixed, one level up. Each group now gets a scoped thread, preserving
order within a group while independent volumes progress in parallel.
* storage: close a volume handle a mount slot replaces
With shared handles, inserting a new Volume over an existing map entry
left the old handle alive in callers — two Volume instances could then
write the same files with separate in-memory indexes. set_volume and
create_volume now close() the replaced handle so its writers fail
closed, matching what unload_volume already does.
* test: key the write hook off a reserved needle id
The hook registry is process-global and keyed by needle id, and other
parallel tests write NeedleId(1) — an unrelated write could enter the
barriers and leave the test's real writer parked forever.
* server: wait out readers for vacuum commit without the store guard
commit_compact try-locked the write lease because callers held the store
write lock, returning COMMIT_WAITS_FOR_READS while a download was in
flight — and the master reads that as a failed vacuum, discarding
completed compaction work. The commit RPC now resolves the handle under
a brief store read and blocks on the volume's lease, matching Go's
CommitCompact under dataFileAccessLock: a slow reader stalls only this
volume's vacuum. The try-lock path stays for callers that still hold the
store lock.
* server: scope the remaining store read guards to volume lookup
Same class as the write-path fix: the batch-delete cookie check and
tombstone, the query read, volume_needle_status, and the local chunk
read all kept the read guard through the needle operation, so slow I/O
there could still stall store writers. Each now resolves the Arc<Volume>
under a short guard; the EC rechecks keep their guards.
* seaweed-volume: fail closed on detached volume handles
A handle resolved under a brief store read can outlive its mount: an
unmount+remount between lookup and lease leaves the Arc pointing at a
closed Volume while a new instance owns the files. Mark the handle in
close() and reject it under the write lease in commit_compact_held,
delete_needle, check_writable, set_read_only_persist and
update_remote_files_held; the tier-down handler also checks the flag
before deleting a remote object its replacement still reads.
* seaweed-volume: drop the store guard before mark-readonly waits on the lease
mark_volume_readonly held a store read across set_read_only_persist, which waits out the volume's leased readers; a slow download would stall every lifecycle operation behind the guard. Resolve the handle under a short read, then mark off the guard; a detached handle now maps to not_found.
* seaweed-volume: retry batch deletes that raced an EC mount
The store guard now drops before delete_needle, so ec.encode can mount shards between the pre-delete recheck and the tombstone; the encode then removes the .dat and the needle resurrects from pre-tombstone shards. Re-check has_ec_volume after each successful delete and return the retriable 503 so the filer requeues onto the EC path.
* seaweed-volume: one write-queue worker per volume
A stalled batch kept later batches for other volumes queued behind it: the single worker awaited process_batch before receiving again. Go's volume_write.go keeps one batching worker per volume, so give each volume its own FIFO worker spawned lazily, bounded by a global concurrent grouped-I/O semaphore and a global queued-request bound; workers exit only when their channel closes, and submit refuses unknown volume ids before spawning.
* seaweed-volume: move the queued-write permit inside the request
submit held the global queued-slot permit across the send and the response wait, so a canceled submit freed the slot while the worker still held the needle; repeated cancels could overflow the bound. The request now owns the permit until the worker drops it.
* seaweed-volume: retire write workers whose volume left the store
worker_for kept a sender per volume forever, so servers that mount and retire volumes accumulated idle workers. A worker now polls while idle and retires once its volume leaves the store: close the channel so racing senders fail fast, drain accepted writes, then release the map slot — a replacement worker can never start while this one still holds queued writes, so same-volume FIFO survives retirement.
* design: durable volume-id allocation boundary (#11763)
* master: mirror max volume id to a file outside raft (#11763)
The allocation bound lived only in raft state, so a master that lost it
rebuilt the bound from whatever volume servers were connected and could
reissue ids held by offline servers.
NextVolumeId now persists the bound to <mdir>/max_volume_id before the
raft apply, and both raft implementations persist on apply and recovery
so follower mirrors stay current. The file also carries the topology id,
which survives raft state cleanup the same way the snapshot recovery did.
* master: serialize and fully sync the max volume id mirror (#11763)
* master: rewrite the mirror when the topology id changes at the same bound
A same-bound raft apply carrying a new topology id was skipped by the
lastPersistedMaxVolumeId check, leaving the mirrored topology id empty
until the next allocation. Track the mirrored topology id too and
rewrite when either value differs.
* drop the design doc from the PR stack
The reverse index walk dedupes seen NeedleIds with a BoomFilters bloom
filter. Its default kernel is FNV-1, whose final step only XORs the last
input byte into the hash's low bits, so keys sharing the high 7 bytes of
a big-endian NeedleId land on correlated probe positions and false
positives arrive in bursts of hundreds instead of the ~0.001 target
rate. Each false positive hides a live-latest key and inflates the
rebuilt deletion counters, which is what made
TestFastLoadingNeedleMapMetrics flake past its +256 bound in CI
(observed DeletedCount drift of 267).
Swap the kernel for xxhash, which fully diffuses the key bytes; the
observed false-positive count in 300 local runs of the test drops to
max 2 (avg 0.23).
* ec: compare live idx NeedleId sets before encode
Refuse encode when replicas' live .idx NeedleId sets differ. Walk only
live rows (skip deleted and Size<=0) and compare NeedleId, ignoring Size
so MIME/flags drift does not abort a matching id set. Mark the volume
read-only for encode and reject writes after that.
* fix(ec): compare all replicas then pick capacity source
Log matching live idx sets with "replicas are consistent" for the
shell CI check. After readonly, compare every replica's live NeedleId
set, then require the encode source to have FreeVolumeCount>=2.
Empty live-idx vs non-empty refuses encode instead of skipping
superblock stubs. Drop the FileCount rebind after the idx gate.
* volume: mark readonly under the write lock so checked appends finish first
* ec: fix readonly deadlock and restore worker-host source preference
MarkVolumeReadonly held dataFileAccessLock while calling
reopenIdxForWrite, which re-acquires the same mutex and deadlocks
TestMarkVolumeReadonlyCanDelete. Split out reopenIdxForWriteLocked for
callers already holding the lock, matching the existing Locked pattern.
The worker task lost its worker-host source preference when the idx gate
replaced the replica union. Late-bind the source to a replica on the
worker host using only the checked location set, with no FileCount
rebinding. Drop the now-unused helpers, trim comments added by the
earlier commits, and rename tests whose names still claimed convergence
while asserting refusal.
---------
Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
A restarted goraft master wipes its raft state and only learns the
TopologyId after the leader re-admits it and replays the entry. That
trails HTTP readiness by several seconds, so the one-shot check in
assertTopologyIdConsistent raced the propagation (CI run 39199033687:
TopologyId applied on the follower ~2s after the test gave up).
* admin: default to port 12535, keep 23646 as a best-effort compat listener
The default admin HTTP port 23646 derives a worker gRPC port of 33646
(port + GrpcPortOffset), inside the Linux default ephemeral range
(32768+). One of our own startup dials can claim 33646 before the admin
binds it, killing startup. This is the same failure #10209 fixed for
the test port allocator, still reachable via the mini default (#10838)
and via standalone `weed admin` defaults.
Move the default admin port to 12535 (gRPC 22535) everywhere, keeping
the http+10000 convention so worker -admin derivation, the Rust worker
and host:http.grpc addressing keep working:
- weed admin -port, weed mini -admin.port
- weed worker -admin, the plugin worker fallback, Rust worker and
weed-lance-worker defaults
- help text, README, helm values, docker compose, tests
So old workers, bookmarks and scripts keep working, admin also serves
the same handler and the worker gRPC server on the previous default
(-port.legacy / -admin.port.legacy, default 23646, 0 disables). The
legacy gRPC port binds best-effort: it lives in the ephemeral range, so
a failed bind only warns instead of reintroducing the startup failure.
weed mini reserves it early like the primary port and keeps it out of
its own port relocation. grpc.Server tracks every listener handed to
Serve, so the compat listener stops with the primary.
Also warn when an explicitly configured admin port puts the derived
gRPC port at or above the ephemeral floor, the other fix suggested in
the issue.
Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc
off the +10000 convention breaks external workers that derive the admin
gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins
-admin.port on their side; this makes that pin unnecessary.
Fixes#10838
* admin: let services win the legacy ports, close compat listener on failure
Review findings on the compat listeners:
- Do not put the legacy ports in mini's reservedPorts: an explicit
-s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected
or stolen by the early reservation. Mini now checks the finalized port
set and skips only the conflicting compat listener, so HTTP and gRPC
compat degrade independently and the service keeps its port.
- Also skip the legacy HTTP listener when a mini service owns the port:
a wildcard-bound service can still leave loopback bindable, which would
let the compat listener shadow it for loopback clients.
- Close legacyServer's raw listener in the deferred cleanup: Serve may
not have registered it yet when startAdminServer returns early.
- Helm: add admin.legacyPort (default 23646) and expose the legacy
http/grpc ports through the admin Service, container ports and
NetworkPolicy so older workers resolved via the Service keep working.
* helm: only pass -port.legacy when it differs from the binary default
ct install deploys the released image with the new chart, and 4.48 does
not know -port.legacy, so the admin pod crash-loops. Omitting the flag at
the default value is behavior-identical on new binaries (their default
is also 23646) and keeps older releases installable; legacyPort: 0 or a
custom port still renders the flag.
* webdav: delete a folder with its contents
* webdav: guard the recursive delete against stale and confined paths
- propagate stat lookup errors instead of returning an error-bearing
FileInfo: OpenFile and DELETE would otherwise take the recursive
removeAll on a path whose type was never confirmed, wiping a
directory on a transient filer error
- refuse wrappedFs.Rename from or to the served root, matching the
RemoveAll guard and webdav.Dir
- normalize trailing slashes in removeAll so the delete names the
entry itself, and refuse the root for every caller
* webdav: keep file-only deletes shallow and refuse ancestor destinations
- removeAll used by OpenFile and failed-upload cleanup now deletes
non-recursively, so a folder swapped in after the stat check is not
emptied by a pending file delete
- MOVE and COPY whose destination contains the source are refused
before the handler removes the destination, which would otherwise
take the source down with it
* webdav: also refuse a destination inside the source
a MOVE or COPY into its own subtree recurses into the destination it
just created; refuse it with the same guard instead of burning the
recursion budget for a 500
* webdav: allow a shallow copy into the source subtree
a COPY with Depth: 0 creates only the destination collection and never
reads children, so it is safe inside the source; only deep copies and
moves are refused there
* webdav: refuse a copy/move whose source is unreadable
* webdav: cover the missing-source refusal in the guard test
* webdav: 500 on a source lookup failure, 404 only when absent
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Brings in the raft snapshot-recovery fixes: a recovered follower returns
to Follower so it can campaign when the leader goes away (masters on
ephemeral -mdir no longer wedge leaderless), and currentTerm stays
monotonic and persisted through recovery.
* filer: free shared chunks when a folder delete drops the last hard link
doBatchDeleteFolderMetaAndData skipped chunk collection for every hard-linked child, relying on maybeDeleteHardLinks to retire the shared inode record. When the delete removes the last link the record is gone and nobody else will ever free those chunks. After the link callback runs, check whether the record survived; if it did not, queue the entry's chunks for deletion.
* filer: free orphaned hard-link chunks in the replayed delete sweep
The subtree sweep on a replayed directory delete skipped every hard-linked child, so a link that was the inode's last owner leaked its shared chunks. Check the inode record after the store delete; when it is gone, queue the child's chunks like any other swept entry.
* filer: mirror remote-object cleanup in the replayed delete sweep
Children a lagging peer alone still holds under a replayed directory delete were dropped with plain store deletes: their remote objects leaked and lazy remote fetches could resurrect them. The sweep now records event-stamped tombstones and deletes swept non-directory children through maybeDeleteFromRemoteIfUnchanged, which carries the same stat guard the write-back daemon's processDeleteEvent applies plus a modification-time check — a delayed replay cannot erase an object another writer replaced, even a same-size one. A failed remote delete fails the replay so the event is retried instead of orphaning the object. The directory's own tombstone comes from the event pipeline at the event timestamp and covers the subtree. Metadata-only deletes and cross-cluster events keep the origin's semantics.
* filerstore: return real errors from KvGet on MongoDB, ArangoDB, Elastic
These stores mapped every lookup failure to ErrKvNotFound, so a database error looked identical to a missing key: hard-link records could appear gone while links still owned their chunks, and path timestamp reads could treat committed state as absent. Return the underlying error and keep ErrKvNotFound for genuinely missing keys.
* filer: bind remote-object deletes to the observed object version
A guarded stat-then-delete still lets a replacement landing between the two requests go down with the deleted object. Add an optional conditional-delete operation to remote storage clients — S3 If-Match on the live ETag, GCS generation match, Azure IfMatch — and use it everywhere a delete is guarded by a stored stamp: the write-back daemon's delete, rename-source, and pending-rename paths, and the replayed-delete sweep. Backends without it keep the guarded stat-then-delete.
* filer: restamp the remote entry after a metadata-only remote update
Changing Content-Encoding rewrites the object in place on S3 and advances its LastModified, while the recorded stamp keeps the old mtime. A version-checked delete would then mistake the object for a replacement and keep it forever. Re-stat after UpdateFileMetadata so the saved stamp describes the live object.
* filer: mirror the remote-delete ownership checks in the replayed sweep
The sweep deleted swept children's remote objects on the listing's snapshot
alone. Two gaps: a remote-only entry that took the path since — one a commit
record may not cover, such as a lazy fetch materialized from the remote —
reads the object as its only copy, and an entry removed while its rename
copy was pending leaves the object at the source key orphaned. The per-child
delete now runs under the child's entry lock with the commit-time check
repeated, keeps a remote-only entry that holds the path, and releases the
pending rename's source under the same ownership and stamp conditions the
write-back daemon's releasePendingSource applies.
* remote.sync: restamp only the object the metadata update touched
The live stat after UpdateFileMetadata was saved unconditionally, so a
replacement written between the update and the stat stamped the entry with
an object it never described — and a later delete would then pass the
freshness guard on the replacement's data. The restamp now requires the live
object to still match the stored stamp on size and ETag; a different object
keeps the old stamp, which the delete guard reads as replaced and keeps.
* filer: skip unbudgeted reader-cache test on 32-bit
TestReaderCacheUnboundedWithoutBudget proves the reader cache has no
implicit memory budget by keeping 96 concurrent 4 MiB downloads in flight
(>256 MiB). On linux/386 the weed/filer test binary already runs near the
4 GiB address-space limit, so the burst reliably crashes it with
"out of memory" mid-suite.
* filer: bail out of a swept child lock instead of waiting on it
Replay already holds the event's path locks when the sweep reaches a
descendant. A local rename can hold that descendant while waiting on one
of the replayed paths, so blocking on the child lock deadlocks both. The
sweep now tries the lock and reports ErrLockContended — transient, so the
replication retry replays the event once the local write settles.
* remote: report the stamp a metadata update produced
A metadata write can rewrite the object it touches — S3 changes
Content-Encoding through an in-place copy, which can mint a new ETag —
so restamping from a fresh stat can reject the very object the update
produced, or describe a replacement that landed in between.
UpdateFileMetadata now returns the RemoteEntry of the object it wrote
(S3 from the copy result, GCS from the update attributes, Azure from a
post-update stat); nil means the object was untouched and callers keep
the stored stamp.
* remote: restamp Azure from the metadata write's own etag
* iam: scope CreateServiceAccount's ParentUser by policy resource
AuthorizeIamAction evaluated the iam:CreateServiceAccount grant against a
hardcoded arn:aws:iam:::* resource, so the policy's Resource never had a
say; #11218 then bound the target to the caller outright. Evaluate
self-target actions against the target identity's ARN instead: a wildcard
resource keeps the deliberate any-target grant, a scoped resource confines
issuance to the named parent identity, anything else is denied.
buildResourceARN gains the ARN pass-through buildS3ResourceArn already has,
so iam: ARNs reach the policy engine intact on the attached-policies route.
Closes#11735
* ci: retrigger after unrelated weed/storage metrics flake
* iam: fold IAM-ARN matching into awsIAMMatch, drop debug test
- remove zz_debug_test.go (local debugging leftover)
- fold awsIAMMatchResource into awsIAMMatch: the arn:aws:iam:: value
prefix selects case-sensitive matching, so no separate name is needed
- trim comments
---------
Co-authored-by: batonogov <batonogov@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: resume into the directory a listing marker names
* s3: keep delimited marker resumes exclusive
* s3: descend a marker-named directory without repeating its key
* s3: emit marker-named dir objects but keep internal folders hidden
* webdav: create a folder under its own name
* webdav: reuse NormalizePath for the MKCOL entry split
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The worker fix reads the source's TTL destroy time through
ReadVolumeFileStatus, which returns the in-memory volume_info. Go's
SaveVolumeInfo stamps volumeInfo.ExpireAtSec before writing; Rust's
save_vif wrote it only to the serialized file, so a Rust source kept
reporting the stale creation-time value and the EC .vif would carry a
wrong expiry. Stamp the live volume_info first, mirroring Go and
save_volume_info.
* sts: recognize issuer-scoped parent user ids
ComputeParentUser output has a fixed shape (base64url SHA-256); let
callers distinguish it from bare subject strings.
* s3tables: keep legacy owner re-anchoring off local and scoped account ids
Re-anchoring a stored bare subject under the single trusted issuer must
only apply to strings that could actually be a legacy OIDC subject. A
stored owner naming the reserved or a configured local account, or an
already issuer-scoped id, can otherwise be claimed by any caller whose
token asserts that string as sub.
* s3tables: re-anchor legacy policy principals in map and array forms
A stored bare subject can appear as {"AWS": sub} or inside a principal
list, not only as a top-level string; re-anchoring only the string form
left legacy Deny statements unable to match the caller.
* s3tables: keep built-in legacy exclusions independent of the account checker
A missing local-account predicate must not disable re-anchoring entirely:
the reserved accounts, ARN-shaped values, and scoped-id shapes are never
OIDC subjects regardless, while legacy Deny statements for plain subjects
still need to resolve so a denied caller cannot fall through to an
identity grant.
* s3tables: exercise legacy Deny against an owner grant
The previous case denied the caller regardless of whether the statement
principal matched; make the caller the owner so a missed match shows.
Cover the array Deny form alongside the AWS map.
* s3api: recheck condition under lock when routed filer ignores condition_alt_key
A filer that predates condition_alt_key silently drops the field and
evaluates a suspended-versioning condition against the empty null slot
alone, so a matching If-Match still gets a false 412. Recheck under the
object lock instead of trusting that response: the lock path resolves the
same entry the read path serves. Genuine precondition failures still
return 412 from the recheck.
* s3api: cover suspended If-Match against a filer ignoring condition_alt_key
* s3api: resend against the resolved entry instead of the lock-path recheck
Rechecking under the gateway object lock does not serialize against
routed writers, which commit under the filer's path lock; a concurrent
routed PUT could land between the recheck and the create and be
overwritten with a stale precondition. Resend the same transaction with
condition_key set to the resolved version file so the condition and the
write stay under the same filer lock. When the resolved entry is the
null slot, a delete marker, or unresolvable, the resend targets the
object path and fails closed.
* s3api: resolve the target version in tag-condition policy lookups
* s3api: apply tag-conditioned policy to object deletes
* s3api: apply tag-conditioned policy to object ACL reads and writes
* s3api: apply tag-conditioned policy to retention and legal hold
* s3api: cover tag-conditioned delete, ACL and object-lock rechecks
* s3api: let tag-conditioned grants redecide denied batch deletes
* s3api: resolve the tagged version in delete policy tests
readChunkRange sent the volume lookup's http:// URL through a bare
http.Client, so with [https.client]/[https.volume] configured every read of
an SSE-KMS object larger than one chunk (and every ranged read through this
path) failed with "client sent an HTTP request to an HTTPS server" and the
GET returned 500. Normalise the URL's scheme and use util_http's global
client, which carries the [https.client] TLS configuration, as the other
volume reads already do. Drop the now-unused volumeServerHTTPClient.
Fixes#11799
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: map PRECONDITION_FAILED to a sentinel error
A conditional CreateEntry that the owner's path lock refuses comes back as
FilerError_PRECONDITION_FAILED; mapping it to ErrPreconditionFailed lets the
S3 gateway tell a rejected precondition apart from a transport failure and
report 412 instead of retrying the write.
* s3api: make the ambiguous-PUT recovery create conditional under the lock
createAfterAmbiguousRoute verified state and then issued an unconditional
create, so a write landing between the lookup and the create was still
clobbered by the stale re-commit (issue #11631). The recovery create now
carries a WriteCondition — IF_NOT_EXISTS for a proven-absent entry,
IF_ENTRY_EQUAL for a directory it may need to write onto — which the owner
filer evaluates under its per-path lock, making the check and the write
atomic. A refused precondition loops back to re-read the state instead of
overwriting whatever won the race.
* s3api: do not fail over a routed transaction past a possibly-sent request
withFilerClientFailover retried the callback on the next filer on any
transport error, so an ObjectTransaction whose response was lost could be
re-applied over a write that committed in between (issue #11631). A gRPC
error cannot distinguish "never sent" from "committed", but channel
readiness can: a non-replayable callback now runs only on a connection that
just reached READY, and failover continues only over channels that never
came up — the one failure mode that provably sent nothing.
* s3api: run ambiguous-route recovery for empty PUTs too
The recovery was gated on a nonempty chunk list, so a zero-byte PUT whose
routed transaction lost its response still fell back to an unconditional
create that could overwrite a newer write and delete its chunks. Chunk-count
mismatch on an empty upload list is enough to distinguish a superseding
entry from this PUT's — there is nothing to save by skipping the check.
* s3api: keep conditional recovery creates off the replay path
The conditional create in the ambiguous-PUT fallback rode the replayable
WithFilerClient failover, so a committed create whose response was lost
could be re-sent to another filer — where a delete in between would let
IF_NOT_EXISTS pass again and resurrect an entry pointing at reclaimed
chunks. Conditional creates now run through the non-replayable failover;
an ambiguous result falls through to confirmCreateLanded, which resolves
the committed entry rather than resending.
* s3api: do not mark a filer unhealthy for an answered request
A reply that reached the callback — including an application-level failure
like a refused write precondition — proves the filer is alive. Checking
received before the health bookkeeping keeps definitive application errors
from tripping the failure circuit or flagging a live routed owner as
unreachable.
* s3api: do not claim an empty stored entry as an ambiguous PUT's own
Two chunkless entries compare equal under sameFileChunks, so an older
empty object satisfies "the route landed" even though it predates this
write — recovery would return success while leaving the request's
Content-Type and metadata unapplied. A landed PUT stores the entry
exactly as built, so require mime/size/md5/content and the extended
attributes to match before claiming it; anything else stays uncertain
and the client retries.
* s3api: prove ambiguous empty PUTs by write id and pin recovery to the owner
- Conditional recovery creates are sent to the routed owner alone instead of failing over: the condition only serializes with routed writes under the owner's per-path lock, and another filer's forwarding is only as correct as its ring view. An unreachable owner now yields an uncertain result rather than a create under a different serialization boundary.
- A proven-absent path no longer satisfies an empty, chunkless PUT: the routed commit may have landed and been deleted, and there are no chunks to verify, so IF_NOT_EXISTS could resurrect a removed object. Recovery returns uncertain for the client to retry.
- Chunkless identity moves from metadata equality to a per-write id (x-seaweedfs-write-id) stamped into Extended: an identical older object can match mime/md5/content but not the id. Entries without this write's id are not claimed, and confirmCreateLanded can now positively confirm a committed empty object instead of always reporting failure.
* filer: shrink zip spool threshold in tests to fix 32-bit OOM
TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate
~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the
weed/filer test binary already runs near the 4 GiB address-space limit, so
these allocations reliably crash it with "out of memory" mid-suite.
Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which
exercises the same memory-vs-temp-file spool paths with KiB-sized buffers.
* s3 lifecycle: update stale runShard and WalkerFunc call sites in tests
Recent master changes added a walks param to runShard (#11797) and made WalkerFunc take []int, but replay_view_test.go and run_bounded_test.go still use the old signatures, breaking go vet and test builds on merge.
* fs.meta.load: wait for in-flight entry creation before returning
On io.EOF and error paths the command returned while CreateEntry
goroutines were still running, so trailing entries could be silently
dropped once the shell exited. Wait for them on every exit path and
guard the shared error with a mutex.
* fs.meta.load: surface worker errors on read/unmarshal failure paths
The ReadFull and proto.Unmarshal error paths waited for in-flight
CreateEntry workers but returned only the local error, hiding a failed
create from the caller. Check loadErr after wg.Wait() like the other
exit paths so the worker error is reported.
Extract the load loop into loadEntries so the concurrency and error
propagation can be exercised by unit tests, and add regression tests
covering delayed creates at EOF and on corrupt input plus worker error
reporting.
* s3: count stored versions as bucket contents
A <key>.versions directory holding version entries is user data; skipping
it let bucketHasUserObjects report an empty bucket while the recursive
delete dropped every stored version.
* test: rename stale reserved-folders case in bucket delete test
.versions directories are now descended into, so the case only holds
because the folder is empty.
* s3api: hydrate runtime policies on pushed identity upserts
PutIdentity propagation (used on every IAM write when the credential
store has a master client) carries the persisted identity proto. That
proto records inline policies only as flattened legacy Actions; their
runtime __inline_policy__ names live in the policy store and were
dropped by the merge. For the interval until the next full reload, an
identity whose policy denied e.g. DeleteObject routed to legacy
CanDo, where "Write:bucket" authorizes deletes.
Hydrate the pushed identity the same way a full load does, so the
merged state always carries its runtime policy names and documents.
* s3api: hydrate runtime policies on pushed group upserts
PutGroup propagation has the same gap as identity pushes: the wire
proto does not carry the group's runtime __inline_group_policy__ names,
so members' attached-policy check lost them until the next reload.
* s3api: attach only runtime inline policy names on pushes
Listing the pushed identity's or group's own inline policies is enough
to keep it on the attached-policy route; a whole-store hydration per
pushed proto is not. Pushes now never write iam.policies, so a stale
read can no longer overwrite a newer committed policy document.
* s3api: carry inline policy documents on pushed upserts
Attaching only runtime policy names left a window where a pushed identity or group referenced a document the local cache had never loaded, denying allowed requests until the next reload. Fetch the pushed entity own inline documents and publish them with the update; reads stay scoped to the pushed entity rather than the whole store, and inline runtime names are only ever populated from this store so a read cannot install a document older than the committed one. Resync the IAM manager only when a push actually installs documents.
* s3api: order pushed inline policy reads against their installs
A pushed UpsertIdentity or PutGroup read inline policy documents from
the credential store and installed them without holding the ordering
guard that full reloads already take, so a delayed push could overwrite
a newer committed document with its stale read and restore a revoked
grant until the next update or reload.
Pushes now hold reloadMu across the store read and the install, the same
snapshot-through-commit serialization the credential-manager load uses.
MergeS3ApiConfiguration also applies config.Policies to the live
iam.policies at commit instead of swapping in its earlier snapshot, so a
policy installed while a merge prepared (a pushed inline document or a
PutPolicy update) is no longer discarded.
* s3 lifecycle: replay only the rules that fit the retention window
runShard handed drainShardEvents the full snapshot, so rules the retention
window assigns to the walk were also matched during meta-log replay. Their
matches are not due yet, and drainShardEvents freezes the cursor at the first
event with a not-yet-due match. With a 3-day and a 30-day expiration rule on one
bucket and RetentionWindow of 8 days, every shard's cursor stopped at the first
30-day object's PUT and stayed there until that object came due. Each pass
then re-read the log from that point and re-dispatched every 3-day expiration
already handled, which surfaces as NOOP_RESOLVED dispatches, a growing
cursor_lag_max, and passes that still report status=ok.
Snapshot.ReplayView drops the replay-kind actions promoted to the walk and
keeps every other action as compiled, and the drain uses it. Promoted rules
are served by the walk as before. RulesForShard, PromotedHash and ReplayView
share one isPromoted predicate. With RetentionWindow unset no active rule is
promoted, so behaviour without a configured window is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: scale the replay-view test with DaysToDuration
The test took its retention window from DaysToDuration but dated its PUTs
in hours, so under the s3tests build tag (one lifecycle day = 10s) every
event fell before the seeded cursor and replay never ran. Express the
event ages in DaysToDuration too, so the test exercises replay at both
scales.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: rehabilitate scan_only mode in the replay drain view
ReplayView shared the base CompiledAction pointers, so a replay-kind
rule locked into ModeScanOnly by a prior compile stayed invisible to
router.Route's event_driven gate even when the runtime retention window
kept it in the replay partition - and it is absent from the walk view
too, so the rule would be served by neither.
Clone non-promoted replay-kind actions the way RulesForShard does
(Mode forced to event_driven), reactivating the scan_only case whose
bootstrap rendezvous already completed, while preserving the base
active bit for genuinely pre-bootstrap actions. Disabled actions are
excluded for parity with the other views, and DESIGN.md now documents
the view.
* s3 lifecycle: keep drain activation observable, require walker with retention
ReplayView now shares pointers for event-driven actions so a mid-pass
MarkActive flip remains visible to the drain; only a scan_only-locked
action is cloned to event_driven, and reactivated only when its
bootstrap rendezvous completed (tracked via a new bootstrapComplete
field on CompiledAction).
validate() rejects RetentionWindow without Walker: promoted rules
would lose their only dispatch path while the drain cursor still
advanced past their events.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3 lifecycle: share the daily bucket walk across shards
The daily pass invoked the walker once per shard, and every shard's
walk listed the full bucket tree while keeping only its own 1/16 of
the entries — 16 full listings per pass by default (#11791).
Run now groups the shards by the walk branch they'll take (recovery
vs steady, using the same cursor data and hash checks runShard applies)
and starts one shared walk per group, so a pass lists each bucket at
most twice. WalkerFunc and WalkBuckets take a shard set instead of a
single shard. runShard keeps its own decision and waits on its group's
walk; a shard whose freshly-loaded cursor disagrees with the plan falls
back to a per-shard walk, so mid-pass cursor rewrites can't under-cover
a shard.
* s3 lifecycle: skip the .versions/ pre-pass on non-versioned buckets
walkBucketTree listed every directory twice — once looking for
.versions/ dirs, once for regular entries. On a bucket that never had
versioning enabled the first pass can only find coincidentally-named
user folders, so the extra listing is pure cost (#11791).
FilerListFunc now takes the walk view and consults
Snapshot.BucketVersioned; non-versioned buckets skip pass 1 entirely.
A *-suffixed .versions user folder on such a bucket is recursed into
as a regular directory so real objects inside still surface; a nil
view keeps the two-pass behavior.
* s3 lifecycle: prune walk directories to active rule prefixes
The daily walk listed every directory under a bucket even when no
rule prefix could reach it. Collect the active rule prefixes of each
bucket from the walk view (new Snapshot.BucketPrefixes) and skip
recursing into directories whose keys can never match; a bucket
with no active actions is skipped entirely, saving the listing.
.uploads stays in scope regardless of prefixes because AbortMPU
rules match DestKey, and .versions expansion is pruned only
conservatively (the logical key is a string-prefix of the dir key).
* s3 lifecycle: join planned walks before Run returns
planWalks starts a shared walk goroutine per group, but a shard whose
authoritative cursor load then fails exits runShard without waiting on
it. Run returned while the walk still listed or dispatched deletes
against connections the caller closes on return. Join every planned
walk after the shard waitgroup.
* s3 lifecycle: treat object-locked buckets as versioned in the walk
Object Lock requires versioning and every writer sets both extended
attributes, but IsBucketVersioned only read ExtVersioningKey. A locked
bucket missing that attribute would have its .versions dirs walked as
ordinary folders, hiding version siblings from lifecycle. Classify
ExtObjectLockEnabledKey as versioned too.
* helm: honor placement values in the bucket creation hook
The post-install/post-upgrade bucket Job ignored nodeSelector,
tolerations and affinity, so it could land on nodes the operator
kept clear, or fail to schedule at all on a fully tainted cluster.
The hook now accepts nodeSelector, tolerations and affinity under
each createBucketsHook (filer.s3, s3, allInOne.s3), read from the
same values section that supplies createBuckets. Unset nodeSelector
and tolerations fall back to the workload the hook already borrows
its security context from; affinity is explicit-only since the
anti-affinity rules of a long-lived server do not suit a one-off
Job.
* helm: honor placement values in the volume resize hook
The pre-install/pre-upgrade resize Job inherited nothing from the
volume server, so on a cluster whose storage nodes are all tainted
the Job could not be scheduled and the upgrade timed out.
The hook now falls back to the volume server's nodeSelector and
tolerations, and accepts explicit nodeSelector, tolerations and
affinity under volume.resizeHook. Affinity stays explicit-only: the
default podAntiAffinity against component=volume would keep the Job
off every node already running a volume pod, which is most of them
during an upgrade. Placement strings are rendered through tpl with
$volumeName bound, same as the StatefulSet.
* mini: stop forcing -dataCenter onto the embedded S3 gateway
-dataCenter names the embedded volume server's data center, but it was
also copied into the S3 gateway options, overwriting an explicit
-s3.dataCenter and making every S3 write carry an explicit dataCenter
pin in its assign request. That pin outranks fs.configure dataCenter
rules, so a mini node in one DC could never steer S3 writes to volume
servers in another.
With the copy removed, assign requests carry no DC by default: the
filer's own -dataCenter still provides the write fallback, and
fs.configure rules now apply to S3 traffic. -s3.dataCenter still pins
reads and writes when set explicitly.
* server: give embedded s3/sftp gateways their own -dataCenter flags
-dataCenter names the embedded volume server's data center, but it was
also copied into the S3 and SFTP gateway options, and neither gateway
had a flag of its own to override it. The injected DC lands in every
S3 assign request as an explicit pin, outranking fs.configure
dataCenter rules.
Add -s3.dataCenter and -sftp.dataCenter (same names as weed filer) and
stop copying -dataCenter into them, so gateway writes go through the
filer's rules and defaults unless a gateway DC is set explicitly.
* filer: stop defaulting embedded gateway dataCenter to -dataCenter
The embedded S3 and SFTP gateways inherited -dataCenter whenever their
own flag was unset. That made every S3 assign request carry the DC as
an explicit pin, outranking fs.configure dataCenter rules, and left no
way to let rules steer writes while -dataCenter was set.
Let -s3.dataCenter and -sftp.dataCenter stay empty unless set, matching
standalone weed s3 and weed sftp. The filer's own -dataCenter still
provides the write fallback.
2026-10-17 12:38:35 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sftp: check delete permission on an existing rename target
renameEntry authorized only the source path and the target directory,
so renaming over a file with a more restrictive ACL destroyed it even
though Remove on the same file was denied.
* sftp: deny a rename that would strand a carved-out ACL
A rename moves data without moving its path-based ACL: renaming an
ancestor directory left a more specific read-only entry behind, so the
relocated file fell under a permissive ancestor ACL and became
writable. Deny the rename when a configured ACL under the source would
grant less write/delete access than the relocated path would get.
* sftp: cover inherited and destination-side ACLs on rename
The rename ACL check compared only configured paths under the source,
so it missed restrictions inherited from ancestors above the source and
deeper grants configured under the destination. It now compares the
moved root plus every configured path under either tree mapped to its
source counterpart, and skips paths whose entries do not actually move.
The target delete check no longer depends on an existence lookup, so a
protected target created during a rename cannot be overwritten; a no-op
rename skips it. Root bypasses the comparison like every other check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sftp: fail closed on rename ACL lookups and restore new-name renames
- checkRenameKeepsAcl returns lookup errors instead of treating them as a missing entry, so a transient filer failure cannot skip the ACL comparison; it also compares read and list gains, not only write and delete, and evaluates each pair with the moved entry's own IsDirectory.
- The target delete check now distinguishes an existing target (still requires delete) from a fresh name (no delete needed) while still enforcing a configured ACL on the target name itself, so a concurrent create cannot slip a protected target under the rename.
* filer: add fail_if_target_exists option to AtomicRenameEntry
Callers that authorize a rename while the target is absent cannot see a
target created in the gap before the move runs. The flag makes the filer
reject the rename when the target exists, checked under the rename path
locks that CreateEntry on the same filer also takes.
* sftp: guard rename against concurrent targets and file-child ACL gains
A rename onto a name that inherits its directory's ACL still needs no
delete access, but now asks the filer to fail if a target appears before
the move, so a concurrently created file cannot be overwritten without
delete permission. Moved directories are also compared in the file ACL
shape, where list no longer counts as read, so their file children cannot
gain read access through the relocation.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec: carry the source volume's TTL expiry into the worker-written .vif
ReadVolumeFileStatus now reports the volume's ExpireAtSec (last write +
TTL), and the admin worker stamps it into the .vif it distributes with
the shards. Previously every worker-encoded .vif had expireAtSec 0, so
EC volumes of TTL collections never expired.
* ec: report expiry from the Rust volume server and cover the .vif carry
The seaweed-volume drop-in server omitted expire_at_sec from its
ReadVolumeFileStatus response, so worker-encoded volumes sourced from it
lost the TTL. Fall back to the nested VolumeInfo expiry for servers that
predate the top-level field, and pin the nonzero/zero carry with a test.
TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate
~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the
weed/filer test binary already runs near the 4 GiB address-space limit, so
these allocations reliably crash it with "out of memory" mid-suite.
Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which
exercises the same memory-vs-temp-file spool paths with KiB-sized buffers.
admin: fix Create Bucket submitting twice
admin.js and the Buckets page template both bound a submit listener to #createBucketForm, so one click sent POST /api/s3/buckets twice. Keep only the page handler.
CommitCompact holds the volume's write lock for the whole commit, and
reads of that volume wait on it. Volumes compacted together tend to
finish together, so their commits land at once and their fsyncs
contend: each commit's lock hold grows with how many run alongside it,
even when the disk is far from its throughput limit.
Add -compactionMaxParallelCommits (volume.compactionMaxParallelCommits
for server/mini). A commit waits for a slot before CommitCompactVolume,
so a queued commit holds no volume lock and reads keep flowing. The
wait counts against the caller's deadline and returns the context
error if it expires; a slot granted to an already-expired context is
handed back, so a request the caller gave up on never starts a commit,
and maintenance mode is checked again once the slot is held.
The slot is released whether the commit succeeds or fails. Compaction
itself is not gated, so -maxParallelVacuumPerServer still sets reclaim
throughput. 0, the default, keeps today's behaviour.
The limit sits in the volume server so it covers every commit caller:
the master's vacuum sweep, the vacuum worker task and volume.tier.compact.
The Rust volume server mirrors the flag and behaviour: queued commits
wait on a semaphore keyed to the grpc-timeout deadline, hold no store
lock, recheck maintenance mode once a permit is held, and release it
on every exit path.
The "<dir>/" existence probe emitted the empty directory as a
FolderMimeType Contents entry, so delimited listings returned a fake
readable key that breaks clients treating every key as an object.
Delimited probes now report the directory as a CommonPrefix -- which
the probe accepts for existence -- while the recursive (no delimiter)
probe keeps the directory marker.
A shard held on two disks of one server produced two identical live
ShardRefs, so resolution reported "ambiguous" even for a fully
node-qualified -shardId. Ambiguity now means the shard lives on more
than one node; duplicate refs on the same node resolve to it.
* fix(topology): capacity queries no longer create empty disk-type entries
AvailableSpaceFor and CapacityFor called getOrCreateDisk, so asking a
node whether it had room for a disk type it lacks created a zeroed
DiskUsageCounts entry for that type. Volume growth walks every data
center, rack and data node, and ToDataNodeInfo publishes diskUsages, so
every node ended up advertising an empty disk (MaxVolumeCount 0,
DiskId 0) of every type ever grown: HDD-only servers showed an "ssd"
disk and SSD-only servers an "hdd" disk.
Read the counters through a non-creating getDisk; a missing entry
answers 0, the same numbers as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq
* fix(admin): don't let an empty disk-type entry shadow a node's real disk
UpdateTopology keys disks by node and disk id. The master published an
empty entry (no capacity, nothing on it, DiskId 0) for every disk type a
capacity check had asked about, and it collided with the node's real
disk 0; map iteration order decided which survived. An SSD-only node
intermittently showed an empty HDD disk, and an HDD node lost its disk
-- and with it its volumes from the volume index, hence "Could not find
volume N on source server" in balance detection.
IsShadowedEmptyDiskEntry matches an empty entry whose disk id another
DiskInfo of the node, one with capacity or data, already uses.
UpdateTopology and CountTopologyResources skip it. A lone empty entry
is still an empty disk of unknown capacity and is kept, as
GetDisksWithEffectiveCapacity expects.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq
* fix(balance): don't seed balance candidates from shadowed empty disk entries
detectForDiskType registered every node with a DiskInfos entry of the
type as a zero-volume candidate. An empty entry the master made up for
a type the node lacks made that node the least-loaded destination, and
its zero max switched the whole disk type to raw-count balancing. The
volume server then refused every move: "no space left hdd" / "no space
left ssd".
Skip entries IsShadowedEmptyDiskEntry matches when seeding candidates.
A server that holds volumes of the type but reports no max still enters
through the metrics and still triggers the raw-count fallback.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq
* fix(admin,balance): ignore made-up disk-type entries at any disk id
The made-up entry is published with DiskId 0, but it need not collide
with a real disk: when location 0 is unavailable (not reported) the
node's usable disk is location 1. IsShadowedEmptyDiskEntry required a
collision, so such an entry survived both UpdateTopology -- where the
unknown-capacity fallback offers it to planners -- and balance
detection, which planned moves onto it again.
Its disk id cannot tell it apart: a DiskInfo with no volumes, no EC
shards and no per-disk max is published with DiskId 0 whichever disk
it stands for. IsPhantomDiskTypeEntry (renamed, since nothing needs to
be shadowed now) matches an empty entry on any node that has a disk of
another type with capacity or data. A real disk looks the same only
while it offers no volume slot, when it can take no volume either. A
node whose entries are all empty keeps them as disks of unknown
capacity, as before.
Reported by review on #11762.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq
* test(balance): assert where the real disks land in the empty-entry test
Check through GetNodeDisks that each server's real disk sits at the
subtest's disk id before planning. The id-1 case already landed on
physical disk 1 (SplitByPhysicalDisk folds all-zero record ids onto the
entry's DiskId), so this only makes the setup explicit; on the first
revision of the fix it shows the real disk at 1 beside the made-up
entry at 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq
* fix(topology,balance,ec): don't publish or trust usage-only disk-type entries
ToDataNodeInfo started from the raw disk-usage counters, so a node that
was asked for capacity of a disk type it lacks kept advertising an empty
DiskInfo for that type even though no such disk was ever reported. Skip
usage entries that have no disk child and hold all-zero counters.
The same stale entries still misled planners that read DiskInfos
directly: the balance plugin's move precondition treated a phantom entry
as proof a target has that disk type, and the EC balancer counted it as
a matching disk. Filter them with IsPhantomDiskTypeEntry.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-16 23:41:26 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep the gateway up when the Iceberg or Lance listener fails to bind
The optional catalog servers ran glog.Fatalf on a bind error, so an
occupied port (e.g. the default Lance port 9101 taken by another process)
killed the entire S3 server. Log the error and run without the auxiliary
listener instead; the S3 API itself still fails hard on bind errors.
* mini: only probe and advertise catalogs whose listeners bound
With aux listener failures no longer fatal, the configured catalog port may
answer as an unrelated process; mini's readiness probe accepts any HTTP
response and would announce that foreign service as the catalog. Record the
bind outcome on S3Options (final by the time the S3 port serves, since the
aux binds happen first) and gate mini's readiness wait and admin UI
advertisement on it.
* s3: test that a failed aux listener bind is non-fatal
Hold the Iceberg and Lance ports open and assert both starters return no
server and leave the bound flags false, covering the new nonfatal path.
* command: range the endpoint test table by index
S3Options now holds atomic.Bool flags, so copying a test case trips
go vet's lock-copy check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: verify SigV4 against the request path the client sent
Internal routers that rewrite URL.Path (the Iceberg catalog doubled-/v1
collapse) broke signature verification: verifyV4Signature re-derived the
canonical URI from the rewritten path while the client had signed the
original one, so valid signed catalog requests were rejected. Stash the
client-sent path in the request context at rewrite time and prefer it when
computing pathForSignature.
* iceberg: collapse repeated /v1 prefixes before route matching
Move the doubled-prefix collapse from the catch-all into a middleware so it
also covers paths that match the {prefix} routes (/v1/v1/namespaces would
otherwise resolve "v1" as the table bucket; the name is unreachable anyway
since bucket names require 3-63 chars). Collapse all leading repetitions in
a single pass and re-dispatch once instead of recursing per segment, and
record the original path in context so signed requests still verify.
* iceberg: slice off repeated /v1 prefixes instead of concatenating
Stripping path[len("/v1"):] per iteration reuses the original string
instead of allocating a fresh copy of the remaining path each pass.
* iam: accept AWS action names for listing multipart uploads and parts
PutUserPolicy rejected s3:ListBucketMultipartUploads and
s3:ListMultipartUploadParts as invalid actions, so no inline policy could
grant these operations by their AWS names. Map both to Read, which is the
action their routes are registered under.
Fixes#11738
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm
* test: check each multipart listing action separately
Granting both actions in one policy let a correct mapping for one hide a
wrong mapping for the other. Test each action on its own and assert it
grants only Read.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm
* iam: scope multipart listing grants to their resolved action
A policy naming s3:ListBucketMultipartUploads or s3:ListMultipartUploadParts
projected to a Read:<bucket> grant, which the legacy route check honored for
every object in the bucket. Map the AWS names to dedicated action constants
and let VerifyActionPermission fall back to the resolved request action, so
the grant covers only the listing operation it names.
* iam: narrow legacy ListParts and ListMultipartUploads mappings
Both policy names projected to Write:<bucket>, granting full object writes
on any bucket where listing was intended. They now map to the same scoped
listing actions as their AWS names.
* iam: bind the multipart listing fallback to the listing routes
The legacy-action fallback in VerifyActionPermission resolved the
operation with ResolveS3Action, which reads the promoted authorization
object. For GET /bucket?uploads&prefix=foo/ the prefix becomes the
object, so the request resolved to GetObject and a
ListBucketMultipartUploads grant was denied. Resolving from query
priority also did not match how the router picks a handler.
Only fall back for the two listing routes, identified from the route
vars the same way the router dispatches them, and add the scoped
actions to MapToIdentitiesAction so GetUserPolicy in the standalone
IAM API rebuilds valid policy actions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-16 21:11:10 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: skip seed-signature verification in chunked reader when IAM is off
When no credentials are configured there is nothing to verify streaming
chunk signatures against. Leave the credential unset so the reader decodes
the aws-chunked body and skips per-chunk verification, matching the
unsigned-streaming path. This also fixes SigV4-signed unsigned-streaming
uploads on open gateways, which previously failed in calculateSeedSignature.
setupIam() now sets isAuthEnabled to match the credentials it installs, so
the signature-verification tests still exercise the enabled path.
* s3: decode signed streaming uploads when IAM is disabled
AWS SDK v2 clients (Trino, Spark, modern aws-cli) send PutObject bodies as
STREAMING-AWS4-HMAC-SHA256-PAYLOAD. On a gateway without S3 auth this was
rejected with ErrAuthNotSetup, breaking writes to table buckets and regular
buckets alike. The auth check already short-circuits when IAM is off, so
route both streaming payload types through the chunked reader.
* s3: cover signed trailer streaming uploads without IAM
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: make every -collection flag parse the same way
The filter vocabulary (comma-separated names, wildcards, regexes,
"_default" for the empty-named collection) already existed in
wildcard.CompileCollectionMatcher, but several commands still did their
own exact-match comparison or sent the raw flag value to servers.
fs.mergeVolumes used a literal string compare, so -collection=_default
silently produced an empty merge plan; it now uses the shared matcher
while keeping "*" as the match-all default. volume.fsck and volume.mark
filter through the matcher too, and volume.vacuum resolves the filter
client-side into per-collection and per-volume requests, since no
Collection value scopes the VacuumVolume RPC to the unnamed collection
alone.
Commands that put one concrete collection name on the wire normalize
"_default" to "" (volume.grow, fs.configure, mq.topic.compact,
collection.delete, ec.rebuild, ec.balance), and the tier -volumeId
commands resolve the volume's actual collection from topology first so
patterns and named-collection requests pass the volume server's
collection check.
* shell: spell the default-collection keyword via CollectionDefault
* shell: address review findings on collection filter handling
volume.fsck keeps a set of every volume the master knows before any
filtering, so a filer entry whose volume was excluded by -collection or
-volumeId is skipped instead of reported "volume not found" and deleted
by -reallyDeleteFilerEntries. The collection filter also no longer
scopes the filer walk to /buckets/<collection>: fs.configure can bind a
collection to any path prefix, and a narrower walk could mark live
chunks as orphans under -reallyDeleteFromVolume.
ec.rebuild and ec.balance resolve -collection through the new
wildcard.ExpandCollectionFilter, which passes literal names through even
when the master does not list them — ec.rebuild must be able to recover
a collection whose shards all lost their index and never registered.
fs.configure and mq.topic.compact now reject _default outright: an empty
Collection means "unset" in both the path-rule merge and the filer's
storage-option fallback chain, so the keyword would have been silently
ignored.
The unnamed-collection vacuum sweep keeps the master's low-disk-space
exception: a volume whose every replica is read-only is probed with
VacuumVolumeCheck and stays eligible when a replica reports
DiskSpaceLow, instead of being dropped outright.
* filer: add zip archive extraction helpers
SpoolZipReader makes an upload body seekable for archive/zip,
SanitizeZipMemberName rejects zip-slip and non-regular members, and
CheckZipLimits caps member count and total uncompressed size.
For seaweedfs/seaweedfs#11722
* filer: extract zip members on ?extract=zip upload
POST/PUT to a directory path with ?extract=zip (or X-Seaweedfs-Extract:
zip) writes each archive member as its own entry, so a bundle of small
files lands in one request instead of one roundtrip each. Members keep
their mtime, mode and mime-by-extension; directories are created;
unsafe names are skipped and reported. extract.keep=true also stores
the archive itself.
For seaweedfs/seaweedfs#11722
* filer: bound zip spool memory by actual bytes
A chunked upload has no Content-Length, so the old size hint let an unbounded body go through io.ReadAll. Read up to the memory threshold plus one byte first, then spill the prefix and the rest to a temp file, capped at ZipExtractMaxBytes total.
* filer: guard zip member size total against overflow
Declared member sizes summing past math.MaxUint64 wrapped the running total and slipped past the 1 TiB cap.
* server: only take the inline fast path at a short first read
uploadReaderToChunks read at most chunkSize bytes, then treated any read below SaveToFilerLimit as the whole file. When SaveToFilerLimit exceeded chunkSize, longer uploads were stored truncated.
* server: resolve storage options per extracted zip member
A member can land under a storage rule more specific than the request path. Re-running detectStorageOption0 per destination applies that rule's collection, replication, TTL, filename limit, and read-only guard to files and directories alike, while explicit query overrides still win.
* server: accept raw request body for POST extract uploads
Every POST was forced through MultipartReader, so a raw zip body always got a 400. Only parse multipart when the Content-Type is actually multipart.
* server: extract zip directories before files and stop on cancel
Processing directory members first keeps their stored mtime/mode from being overwritten by parents auto-created for an earlier file member. Checking ctx.Err() between members keeps a disconnected client from driving thousands more writes.
* server: assert stored mtime and mode for extracted members
* s3: add zip extraction header constants
* s3: extract zip members on PutObject with extract header
x-amz-meta-snowball-auto-extract or x-seaweedfs-extract: zip on a PutObject fans out the archive to one object per member under the object's directory, reusing putToFiler so chunking, collections, storage rules, versioning, and bucket-default encryption apply to each member. The archive itself is not stored. Explicit SSE request headers are rejected since one encryption config cannot represent independently created member objects.
* s3: bound member fan-out by request cancellation
Stop starting new member writes once the request context is done, matching the filer extract loop.
* filer: bound compressed zip spool and reject non-local member names
The 1 TiB uncompressed limit was also capping the compressed bytes
spooled to the host's temp directory, so a handful of concurrent
extractions could fill the filer's disk before any ZIP validation ran.
SpoolZipReader now takes a separate compressed-size limit (the filer
passes its TUS max upload size) with a 5 GiB default.
SanitizeZipMemberName gains a filepath.IsLocal check: the segment loop
already rejects "..", but CodeQL only treats member names as sanitized
through a recognized guard, and the extra check also covers reserved
Windows names.
* server: honor destination rules for zip members and clamp maxMB
An x-seaweedfs-destination header picks the storage rules for a logical
destination while members still land under the request path, matching
PostHandler. Directory members now also enforce the actual path's
read-only rule instead of relying on option resolution to fail.
maxMB up to 2048 overflowed the int32 chunk size, so an oversized query
value made extraction (and ordinary autoChunk uploads) store members
empty yet report success. The parsing is shared and clamped at 2047 MB.
* s3: scope zip member writes through the per-key checks PutObject applies
The extraction loop bypassed most of what a standalone PutObject would
face: members were written without IAM authorization for their key,
directory members replaced markers without entry lock checks, and lock
headers persisted retention without the s3:PutObjectRetention or
s3:PutObjectLegalHold permissions. authorizeMember now runs IAM,
table-bucket path rules, bucket policy, lock-header validation, and
non-versioned lock enforcement per member key, and directory members
take the same guarded mkdir path as an explicit dir marker.
The archive's own Content-Md5 and x-amz-checksum-* values are verified
while the body spools instead of being silently dropped, members stop
inheriting the archive's Content-Encoding, and non-versioned members
resolve their lifecycle TTL like normal uploads.
* s3: fail malformed lock headers before spooling the archive
Per-member re-validation turned a bad object-lock header into one error
entry per member inside a 200 response. Validate once up front like
PutObjectHandler does, and keep only the dedicated retention/legal-hold
permission checks scoped to each member key.
* filer: return 413 for oversized archives and honor saveInside in extract
Map filer.ErrZipTooLarge to 413 instead of 500 for client-side archive
size violations. Extracted members written under saveInside now store
as inline entry content instead of unreachable volume chunks.
* s3: write extracted members under their own request context
Point member writes at the member key via mux.SetURLVars so SSE-KMS
builds the encryption context from the member ARN, not the archive's.
Map the spool limit error to ErrEntityTooLarge.
* filer: add zip archive extraction helpers
SpoolZipReader makes an upload body seekable for archive/zip,
SanitizeZipMemberName rejects zip-slip and non-regular members, and
CheckZipLimits caps member count and total uncompressed size.
For seaweedfs/seaweedfs#11722
* filer: extract zip members on ?extract=zip upload
POST/PUT to a directory path with ?extract=zip (or X-Seaweedfs-Extract:
zip) writes each archive member as its own entry, so a bundle of small
files lands in one request instead of one roundtrip each. Members keep
their mtime, mode and mime-by-extension; directories are created;
unsafe names are skipped and reported. extract.keep=true also stores
the archive itself.
For seaweedfs/seaweedfs#11722
* filer: bound zip spool memory by actual bytes
A chunked upload has no Content-Length, so the old size hint let an unbounded body go through io.ReadAll. Read up to the memory threshold plus one byte first, then spill the prefix and the rest to a temp file, capped at ZipExtractMaxBytes total.
* filer: guard zip member size total against overflow
Declared member sizes summing past math.MaxUint64 wrapped the running total and slipped past the 1 TiB cap.
* server: only take the inline fast path at a short first read
uploadReaderToChunks read at most chunkSize bytes, then treated any read below SaveToFilerLimit as the whole file. When SaveToFilerLimit exceeded chunkSize, longer uploads were stored truncated.
* server: resolve storage options per extracted zip member
A member can land under a storage rule more specific than the request path. Re-running detectStorageOption0 per destination applies that rule's collection, replication, TTL, filename limit, and read-only guard to files and directories alike, while explicit query overrides still win.
* server: accept raw request body for POST extract uploads
Every POST was forced through MultipartReader, so a raw zip body always got a 400. Only parse multipart when the Content-Type is actually multipart.
* server: extract zip directories before files and stop on cancel
Processing directory members first keeps their stored mtime/mode from being overwritten by parents auto-created for an earlier file member. Checking ctx.Err() between members keeps a disconnected client from driving thousands more writes.
* server: assert stored mtime and mode for extracted members
* filer: bound compressed zip spool and reject non-local member names
The 1 TiB uncompressed limit was also capping the compressed bytes
spooled to the host's temp directory, so a handful of concurrent
extractions could fill the filer's disk before any ZIP validation ran.
SpoolZipReader now takes a separate compressed-size limit (the filer
passes its TUS max upload size) with a 5 GiB default.
SanitizeZipMemberName gains a filepath.IsLocal check: the segment loop
already rejects "..", but CodeQL only treats member names as sanitized
through a recognized guard, and the extra check also covers reserved
Windows names.
* server: honor destination rules for zip members and clamp maxMB
An x-seaweedfs-destination header picks the storage rules for a logical
destination while members still land under the request path, matching
PostHandler. Directory members now also enforce the actual path's
read-only rule instead of relying on option resolution to fail.
maxMB up to 2048 overflowed the int32 chunk size, so an oversized query
value made extraction (and ordinary autoChunk uploads) store members
empty yet report success. The parsing is shared and clamped at 2047 MB.
* filer: return 413 for oversized archives and honor saveInside in extract
Map filer.ErrZipTooLarge to 413 instead of 500 for client-side archive
size violations. Extracted members written under saveInside now store
as inline entry content instead of unreachable volume chunks.
2026-10-16 21:04:54 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer_pb: add request_id to ObjectTransactionRequest
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: replay recorded outcome for a retried ObjectTransaction
A failover replay carries the same request_id; under the same path lock
the owner replays the recorded outcome instead of re-applying the
mutations over whatever superseded them. Only application-level
outcomes are recorded — a lookup error leaves the request free to
re-run.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: stamp routed object transactions with a request id
One id per logical transaction so failover retries replay the recorded
outcome instead of re-applying the mutations.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: scope transaction replay tokens per path and keep them for their TTL
The dedup key includes the lock path so a token reused on a different
object cannot replay this outcome. Records are never evicted early —
an early eviction would let a replay re-apply — so the expiry sweep is
amortized and the map is bounded by the insert rate over the TTL.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound the transaction replay cache at capacity
* filer: do not record indeterminate transaction outcomes
A mutation error can be transient or follow a partial apply; recording
it replays that failure to a retry for the whole TTL instead of letting
the same idempotent mutations converge. Only successful transactions
and precondition failures are definitive outcomes worth replaying.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: replay recorded mutation failures instead of re-applying
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 20:05:57 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep suspended-versioning conditional writes off the routed condition
A suspended PUT writes the null-version slot at the regular path while the
entry the conditional headers evaluate lives under .versions/. The routed
WriteCondition checked the request's etag against that empty slot, so a
correct If-Match failed with 412. Passing uniqueWritePath sends conditional
requests through the lock path, whose precondition resolves the same entry
the read path serves; unconditional writes still route.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: keep the volumes progress label readable at low fill
The dashboard's Volumes and tier-usage cells rendered their text inside the
progress bar, so a low fill left white label text on the light track and the
count was unreadable. Render the label beside the bar instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: keep a minimum width on dashboard progress tracks
A nonwrapping label beside a flex track could squeeze it to zero in a
narrow table, hiding the bar entirely.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 20:03:23 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: let an explicit policy Deny veto fallback and owner allows
Table operations evaluated the table or namespace policy and the bucket
policy as an OR, so an explicit Deny in one policy was overridden by an
Allow in the other. Owner and identity grants also short-circuited
before the resource policy was read. Deny statements are now evaluated
first at both levels, matching AWS explicit-deny semantics.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: fail closed when a resource side has no owner
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:54:46 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: apply destination retention on copies instead of inheriting it
CopyObject carried the source object's retention mode, retain-until date
and legal hold into the destination entry, so a copy onto a bucket with
default COMPLIANCE retention produced an unlocked object that could be
deleted early. Copied objects now take their retention from the copy
request's lock headers or the destination bucket's default, matching
PUT semantics; a versioned self-copy gets the same treatment while an
unversioned metadata-only self-copy keeps the object's existing lock.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: validate lock headers on metadata-only self-copies
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:52:10 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: evaluate tag conditions on copy-source and tagging subresources
Policy evaluation ran without the object entry on several paths, so a
Deny conditioned on s3:ExistingObjectTag never matched:
- CopyObject, UploadPartCopy and RenameObject checked the source key
before its entry was fetched; repeat the check on a synthesized GET
once the entry resolves.
- GetObjectAttributes and Get/Put/DeleteObjectTagging never supplied
the entry; recheck with the resolved entry.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: fail closed on tag-policy lookups and honor tag-conditioned grants
- recheckTagConditionedPolicyWithLookup returned success when the entry
lookup errored, letting a tag-gated Deny slip past a transient miss;
only ErrNotFound still falls through to the handler's own NoSuchKey.
- A read or delete denied in the tagless first phase now gets one more
evaluation with the resolved entry's tags in hand, so an
s3:ExistingObjectTag-conditioned Allow can still grant a copy-source
read or rename. The deferred recheck requires an explicit Allow;
indeterminate results keep the original verdict, and the existence of
the key never leaks past a deny.
- RenameObject now applies the same tag-aware recheck to the source
removal, so a tag-gated Deny on s3:DeleteObject blocks a rename the
way it blocks a direct DELETE.
- GetObjectAttributes evaluates the object-entry policy recheck before
conditional headers, so a 304/412 response can no longer bypass a
tag-based Deny.
* s3: honor identity denies in the tag-aware source recheck
The deferred recheck re-evaluates only the bucket policy, so a bucket
Allow could lift an explicit Deny in the caller's attached or session
policies on a copy/rename source. Apply the same applicable-policy Deny
check the first phase uses.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:50:36 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: scope OIDC-derived account IDs to the token issuer
Table ownership keyed off the bare sub claim, so two identity providers
issuing the same sub collided on one account - a caller could act on
tables owned by a different provider's user. OIDC bearer tokens now
carry the same issuer-namespaced (sub, iss) hash STS sessions already
expose as aws:userid, and getAccountID prefers it over the raw claims.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: keep legacy owners working and scrub client-subject claims
- OIDC bearer tokens without a subject are rejected, and a token-supplied
aws:userid claim is always overwritten with the verified (sub, iss)
hash, so a forged claim can no longer stand in for the principal.
- Owners written before issuer scoping hold a bare subject; permission
checks now re-anchor that subject under the caller's issuer, so the
same (sub, iss) pair keeps its ownership while scoped owners remain
issuer-isolated. The token issuer is asserted into claims alongside
sub and carried through PolicyContext.
* s3tables: anchor legacy subjects to the configured issuer set
* s3: enumerate trusted issuers from registered providers too
* s3: fail closed when the OIDC provider store cannot be listed
trustedOIDCIssuers kept the registered issuers when ListOIDCProviders
failed, presenting a partial set as complete. A deployment trusting one
registered issuer A plus a stored issuer B would, on a listing error,
report only A — the single-issuer shape under which a legacy bare
subject re-anchors — letting a caller from A claim a stored subject
that belongs to B and gain owner access.
With no provider store the registered providers are provably complete,
so keep returning them; with a store, a failed listing means the real
set is unknown, so return none.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:48:32 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: sweep the local subtree on a replayed directory delete
A recursive delete emits only the directory's own key, so peers applied
it with a single DeleteEntry. Children the peer held that the origin had
not listed stayed behind unlisted but still readable, and their needles
were never deleted. Sweep the local children on a replayed directory
delete and drop their chunks when the origin deleted chunks.
* filer: walk the subtree on a replayed directory delete
DeleteFolderChildren removes only direct children in stores keyed by
immediate parent, so the sweep now lists and deletes each descendant
entry itself. Entries the local filer committed after the event survive
— with the directories that hold them — and hard-linked files keep
their chunks for the origin's own events to settle.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: key replayed subtree chunk deletes to each entry's path
The sweep queued every descendant's chunks under the deleted directory's
storage rule and skipped directory entries entirely. Chunks now collect
keyed by their owning path, so per-path rules apply to each file — and a
directory carrying its own chunks (a file promoted by an S3 child) frees
them too.
* filer: fence kept replayed dirs and flush collected chunks on error
A directory kept because a newer descendant survived still needs its
delete timestamp recorded, or a stale replayed create can resurrect
another child under it. And chunks collected for deleted children must
be flushed even when the sweep fails partway — the entries are gone and
a retry cannot list them again. Collection now also runs only after the
entry delete succeeds.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: flush swept chunk references in batches
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:47:20 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: keep volume.tier.compact replicas on the same remote object
The compact cycle ran on a single replica: it deleted the shared remote
object up front and never touched the siblings, leaving their .vif/.idx
pointing at a removed object. It also remounted the working replica into
a writable local volume mid-operation.
Freeze all replicas first, keep the shared object while siblings still
reference it, replicate the new metadata to every replica, and let the
last tiered replica retire the old object.
* storage: surface remote object delete failures from volume destroy
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: keep tier.compact replicas frozen until every copy lands
A delete between the source unfreeze and a sibling's index copy could
land past the copied snapshot, leaving the replica serving deleted
data. Unfreeze all replicas only after every resync completes; a remote
volume's readonly mark is inert, so the freeze adds no extra cost.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: abort volume destroy when remote delete fails
destroyLocked removed local files even when the backend delete errored,
leaving a registered replica with no .dat/.idx and a leaked remote
object. Run the remote delete before local cleanup and bail out on
failure so the volume stays fully usable and the delete can be retried.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: mirror destroy ordering for remote delete failures
Propagate a failed remote-object delete instead of warning past it, and
reinsert the volume into the disk location registry so a still-intact
replica is not unregistered.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: recover partial tier.compact freezes and report held replicas
A failed freeze left already-marked replicas readonly with nothing
changed, and later stage failures returned without saying the replicas
stay frozen or how to resume. Roll back the partial mark when freezing
fails, note in every mid-flight error that replicas remain readonly and
rerun is the recovery path, and report how many replicas a partial
unfreeze released.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: retire abandoned remote objects during tier.compact resync
A resync overwrote each replica's .vif, orphaning its previous remote
object; an interrupted run could leave replicas on different keys, and
only the last tiered replica's object was deleted. Read every replica's
remote file list up front, count holders per object, and delete-before-
copy whichever replica is an object's last holder, so shared and
stranded objects alike are retired instead of leaked.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep collection members registered when destroy fails
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:45:17 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep suspended-versioning conditional writes off the routed condition
A suspended PUT writes the null-version slot at the regular path while the
entry the conditional headers evaluate lives under .versions/. The routed
WriteCondition checked the request's etag against that empty slot, so a
correct If-Match failed with 412. Passing uniqueWritePath sends conditional
requests through the lock path, whose precondition resolves the same entry
the read path serves; unconditional writes still route.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer_pb: add condition_alt_key to ObjectTransactionRequest
* filer: evaluate a transaction condition against a fallback entry
condition_alt_key names the entry consulted when the primary is absent,
so a suspended-versioning write can check 'the null slot if present,
else the latest version' atomically under the lock.
* s3: route suspended-versioning conditional writes with a fallback key
The gateway names the latest version file as condition_alt_key, so the
transaction checks the entry the read path serves — null slot first,
latest version when it is absent — under the same lock that applies
the write. Conditional suspended PUTs route again instead of splitting
check and write across a gateway lock that routed writes skip.
* filer: resolve the condition fallback under the transaction lock
The gateway named the pointed-at version file as condition_alt_key,
resolved before the transaction ran: a concurrent version delete could
move the latest pointer between the gateway read and the locked check,
and a delete-marker latest read as an existing object. The alt key is
now the .versions directory and the filer resolves the latest-version
pointer under the same lock that serializes the write — a delete marker
resolves to absent, and unresolvable pointer state fails closed.
* filer: fail closed on alt-key lookups and skip directory primaries
A transient FindEntry error returning a nil entry read as "versions dir
absent" and let If-None-Match writes pass. Check the error before the
entry. And a bare directory at the object path satisfied the primary
lookup, so the .versions fallback never ran — directories are not the
object, treat them as an absent slot.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: scope directory-as-absent conditions to alt-key callers
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: default allowDeleteBucketNotEmpty to false
DeleteBucket on a non-empty bucket silently destroyed every object,
while AWS returns 409 BucketNotEmpty. Clients like 'aws s3 rb' rely on
that refusal as the 'delete only if empty' safety net.
Deployments that need the recursive delete can opt back in with
-allowDeleteBucketNotEmpty=true.
* test: pin allowDeleteBucketNotEmpty=true on versioning config1
The config1 run relied on the flag's old true default to contrast with
config2's false. Pin it so the two modes stay distinct.
* test: drop redundant allowDeleteBucketNotEmpty=false in s3 integration test
False is the default now, so TestS3DeleteBucketNotEmpty exercises the
out-of-box behavior.
2026-10-15 23:45:36 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: release flock locks before the close-time flush
Release ran the dirty-handle flush before dropping the file
description's flock locks. A flush parked on a dead filer kept a dead
holder's lock in the table, so a restarted process taking flock on the
same file saw "file already locked" until the mount restarted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: finish the blocked-flush release test deterministically
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: let the release test's parked flush talk to a live filer
Marking the handle deleted raced with the flush's unlocked early read
of isDeleted. Pointing the fixture at a live fake filer lets the flush
complete on its own once the handle lock frees, no flag write needed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.8.0 to 1.8.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.8.0...sdk/azidentity/v1.8.2)
---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
dependency-version: 1.8.2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* test: restore loadBucketMetadataFromFiler stub after TestGetBucketMetadata
TestGetBucketMetadata replaced the package-level loadBucketMetadataFromFiler
with a stub that sleeps one second per load and never restored it. Every
test running after it in the same binary paid that sleep on each bucket
metadata cache miss: TestPutObjectUploadACL (~245s) and
TestCopyObjectDestinationACL (~46s) each created a fresh BucketRegistry per
subtest, so the stub fired hundreds of times. In CI this stretched the
Test job past 11 minutes and left it exposed to runner cancellation.
Restore the real loader via t.Cleanup so the slow stub is scoped to the
test that needs it.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* ecbalancer: add Node.SetNoReceive to exclude a node as a move target
A node that has refused a shard copy at runtime (e.g. 'no space left' on the actual disk, issue #11753) can no longer be trusted as a destination for the remainder of a balance run, but it may still hold shards that must move away. SetNoReceive excludes the node from target selection and from receiving-side capacity accounting, while leaving it eligible as a move source.
* ec: re-plan balance moves when a target reports no space left
Moves are planned from a heartbeat snapshot, but concurrent copies let the volume server's real disk state diverge from it, so a planned target can refuse the copy with 'no space left' mid-run and abort the whole balance (issue #11753). Track the refusing targets during execution, mark them no-receive, re-collect the topology, and plan the remaining moves again. The loop terminates because a marked node can never be picked as a target again.
* volume server: return ResourceExhausted when no disk can take an EC shard
A failed EcShardsCopy carried code=Unknown even though the cause is exhausted local capacity; ResourceExhausted describes the refusal and lets clients distinguish it from internal errors. The message text is kept so message-based matching still works (the Rust volume server reports the same text under Internal).
* ec: replay applied moves onto the re-collected snapshot and surface failures
Addresses review on the re-plan loop: a fresh topology read can still describe an applied move while heartbeats lag, so journal each applied move and replay it onto the next snapshot; retry only when every failure was a 'no space left' refusal (an unrelated failure aborts instead of being hidden); and exclude refused nodes' slots from the free-space precheck so a balance with no remaining receiver fails instead of reporting success for an empty plan.
* ec: carry per-disk state when replaying applied moves
A move that landed on an explicit TargetDisk loses that attribution when replayed onto a stale snapshot, leaving the disk's free slots and shard count unchanged; the next plan could pick the same disk, hit another refusal, and over-exclude the node. Replay the shard onto the target disk's entry and move the disk-level counters, and restore the source disk's counters on removal.
* ecbalancer: add Node.SetDiskNoReceive to exclude a single disk as a target
A copy with an explicit disk_id can be refused while the node's other disks still have room; a disk_id 0 refusal already means the server searched every disk, so the scopes differ. disk.noReceive removes that disk from receiving capacity and disk selection while leaving the node and its sibling disks eligible.
* ec: record explicit-disk refusals as disk-scoped no-receive
recordMoveResult now distinguishes the two refusal scopes: TargetDisk>0 marks only (node, disk), while TargetDisk==0 marks the whole node since the server's auto-pick found no usable disk. The free-slot precheck subtracts refused disk slots, and the replan log names refused destinations as node:diskN.
* ecbalancer: skip no-receive disks in pickBestDiskOnNode
The balance phases pick the destination disk through pickBestDiskOnNode, which ranked disks by score only; a refused disk with a low shard count would be re-picked, fail again, and abort the run since no new destination was rejected.
* volume_move: export IsNoSpaceLeftError for move callers
Both the ec.balance replanner and the worker ec_balance task need to recognize the volume server's no-space refusal; keep one matcher next to the move RPCs instead of a private copy in weed/ec.
* worker: retry pinned-disk EC moves with a server-picked disk on no space
ec_balance detection always pins TargetDisk, so a full disk fails the job even when a sibling disk has room. On a no-space refusal to the pinned disk, retry the move once with TargetDisk 0 so the volume server picks a disk on the same node; the planner's node choice stands.
* mini: note mini.options and Docker filer.toml pitfalls in help
Two behaviors of "weed mini" surprise users and are not mentioned in
its help text.
Flags are saved to <dir>/mini.options and applied again on the next
start to every flag that still has its default value. Removing a flag,
or passing it with its default value, therefore keeps the saved value.
The official Docker image ships /etc/seaweedfs/filer.toml, which sets
the leveldb2 store to /data/filerldb2. When filer.toml is found, the
filer ignores the default store dir derived from -dir, so moving -dir
away from /data leaves the metadata in /data. If /data is not
persisted, buckets look empty after the container is recreated.
Only the help text changes.
Signed-off-by: YoungjuKim <fjvbn2003@gmail.com>
* mini: update help notes for computed-only mini.options
---------
Signed-off-by: YoungjuKim <fjvbn2003@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
mini: persist only computed options, and warn when filer store escapes -dir
- saveMiniConfiguration no longer writes command-line flags to
mini.options; the file now only pins values mini computed itself
(currently master.volumeSizeLimitMB), so a dropped flag can no longer
resurrect on the next start. Existing files still apply once and are
then rewritten without the saved flags.
- applyConfigFileOptions skips any flag explicitly provided on the
command line or via WEED_* env vars, so passing a flag always wins
over a saved value, including at its default value.
- mini prints the saved options it applies at startup (secret values
redacted), since quietMiniLogs hides the previous V(1)/V(2) messages.
- warn once at startup when a filer.toml points a local filer store
(leveldb/leveldb2/leveldb3, including path-specific <store>.<id>
sections) outside -dir and -master.dir: the filer skips the
-dir-derived store dir whenever a filer.toml exists, and the metadata
is lost on container recreate if that path is not persisted.
* filer: escape LIKE wildcards in SQL list prefixes
ListDirectoryPrefixedEntries bound prefix+"%" straight into a LIKE
clause, so '_' and '%' in object names (e.g. Langfuse keys like
call_03_...) acted as wildcards: wrong results and a ~40x slower
range scan because (dirhash, name) can no longer be narrowed.
Escape '!'/'%'/'_' in the prefix and declare ESCAPE '!' in the
generated list SQL so every abstract_sql backend (mysql, mysql2,
postgres, postgres2, sqlite) matches literally with identical
semantics. '!' is used instead of '\' because SQLite shares
SqlGenMysql but would reject the MySQL-style '\\' literal, and
'\' would still need escaping itself.
* filer: escape LIKE wildcards in YDB list prefixes
Same wildcard-in-prefix bug as the abstract_sql backends: YQL LIKE
treated '_' and '%' in the prefix as wildcards. Apply the shared
LikePattern helper and declare ESCAPE '!' (YQL allows any escape
char except '%', '_' and '\').
* test: cover literal '_'/'%'/'!' prefixes in sqlite filer listing
End-to-end regression test through SqliteStore against a real
database; runs in CI under the sqlite build tag.
* filer: create parent directories for lazily fetched remote entries
A remote-mounted object first read through the filer was persisted without
its parent directories, so remote.uncache and the remote cache evictor, which
walk down from the mount directory, never reached it: its cached chunks could
not be evicted.
Persist the missing parents with the entry. With those directories in place,
the empty-folder cleaner leaves folders under a remote mount alone: deleting
one removes its whole prefix on the remote, which can still hold objects
that were never read locally.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep remote objects at parent keys and remote prefixes of empty folders
Look up a lazily fetched entry's missing parents through lazy fetch before
creating them, so a remote object stored at a parent's key is materialized
and promoted to a prefix object rather than hidden behind a new directory.
Deleting a folder that is empty in the filer is metadata-only under a remote
mount, both for a non-recursive delete and for the expiry cleanup of empty
parents: the mount is populated lazily, so the folder can still hold remote
objects never read, and removing its remote directory deletes every object
under that prefix. This replaces the empty-folder cleaner's own guard, which
covered only one of those paths.
A metadata-only delete records no remote-deletion tombstone, locally or from
its metadata event: nothing is owed to the remote, so lazy reads stay open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: stop the lazy parent lookup at a store error
Only a parent the store reports as not found is missing; a parent it cannot
read ends the walk, so the lookup does not stat ancestors above it on the
remote.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: stat each missing remote parent once
A lazy fetch that finds k missing parents asked FindEntry for each, and every nested fetch re-walked its own still-missing ancestors, re-stating remote keys already proven absent: 1+k(k+1)/2 remote stats for one first read. The flagged parent lookups now skip their own walk; their entries are created by the same CreateEntry that already persists the chain.
* filer: materialize parents for lazily listed remote entries
maybeLazyListFromRemote persisted children and the listing directory with skipCreateParentDir, so a first access that goes through the list path still left entries unreachable to a walk from the mount directory. Materialize the missing ancestors the same way the lazy fetch does, and let the synced_at directory create its parent chain.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: record the size of a multipart upload saved inside the entry
* filer: guard save-inside multipart append and check Content-MD5
Addressing review feedback:
- Reject a save-inside multipart append when the target is chunk
backed. The buffered content cannot be attached to existing chunks,
so previously the request grew FileSize without storing the appended
bytes, and reads returned zeros for the added range.
- Compare the request Content-MD5 against the buffered content before
saving, matching the regular multipart path, so a bad digest no
longer commits the file.
- Extend the size test to check the saved entry MD5, and cover the
rejected append against chunk-backed and inline targets plus the
bad-digest rejection.
* filer: return 400 for unsupported append modes
The save-inside append rejection and the existing inline append and offset write rejections are deterministic client errors, but the autoChunk default case mapped them to 500. Give them shared constants and return 400 instead, and assert the status in the tests.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Jay HemnaniGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sftp: keep a file's existing bytes when it is opened for write without truncate
* sftp: preserve sparse gaps and refuse unreadable entries when copying existing content
The copy used a single chunk-stream read, which ends at the first sparse
gap: the tail past a hole was dropped, and a remote-only entry with no
local chunks copied zero bytes, so closing the handle uploaded an empty
file. Read through PrepareStreamContent, which fills unstored regions
with zeros, and refuse entries that carry no local bytes at all.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: report a failed destination write from a completed chunk fetch
retriedStreamFetchChunkData broke out of the retry loop on a clean HTTP
end before checking writeErr, so a destination write that failed mid-
stream returned a short write and nil error. Check writeErr first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sftp: cover remote and all-holes sizes when copying existing content
Take the declared size from RemoteEntry when it exceeds FileSize, pad
inline content to the declared size, and treat a chunkless local file as
all holes rather than unreadable. A remote-only entry still refuses the
copy, now checked with a zero FileSize as well.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sftp: size the existing-content copy the way filer.FileSize does
A locally shortened file still names its older, larger remote object;
taking RemoteSize unconditionally padded the copy back out to it.
filer.FileSize only honors the remote size when the remote is newer,
and also covers the chunk-extent size. A remote-only entry still
refuses the copy whenever it points at remote bytes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
LocateEcShardNeedle passed needle.GetActualSize(size, version) to
LocateEcShardNeedleInterval, which calls needle.GetActualSize internally.
This caused header and padding to be added twice to needle size.
Fixes#3864
Co-authored-by: Morteza <m.mohammadi@rahkar.team>
* feat: prefer same-host worker and local EC replica
Admin executor pick ranked idle workers only, so a job whose volume
lived on full-03 could still run on full-01 and pull the whole .dat
across the LAN. Rank workers whose Hello host matches any replica
first (fallback to others if the local worker is full).
Detection named Sources[0] / the lowest server id; execute then
SyncAndSelectBestReplica and could switch the copy source to another
rack. After sync, if a replica on this machine has a complete file
count, copy from that URL so affinity and pull stay on one host.
* feat: pin scheduler jobs to replica-host workers until load K
Reserve prefers the least-loaded replica-host worker and waits while
that load is below 5 instead of spilling to a free remote worker.
K=0 restores immediate spill. Waterfill uses reservation load so four
idle preferred workers each get a job.
* feat: park scheduled jobs on per-worker admin queues
Assign a detection batch to the least-loaded replica-host worker in
one pass, then let ExecutionConcurrency goroutines pop a home-runnable
job or steal when that home queue reaches K. Dispatch no longer holds
a job in reserve-backoff, so a free worker is not blocked by another
home's pin. Tracked jobs stay pending/queued and record the home
worker for the admin UI.
* fix: rehome queued jobs when the home worker dies
A live home still pins below K. A home missing from live executors
is stolen immediately onto a free worker so a short orphaned queue
does not sit until the dispatch window ends.
* feat: wait before stealing a dead home worker queue
Orphaned queues are held for 30s so a worker reconnect blink does
not dump jobs onto other machines. After the grace, steal without K
as before. Zero disables the wait. A home that comes back during
grace keeps its queue.
* fix: late-bind EC from to worker hostname
After replicas are consistent (equal FileCount), rebind encode source
to a volume server on the same hostname as worker -address. Any port
on that host is local. Diverged FileCount still uses the max replica.
* feat: assign homeless jobs to global FIFO
Jobs whose replica hostname has no registered worker skip pin and orphan grace.
Free workers take own home, then steal at K/orphan, then leftover FIFO.
* fix: take homeless FIFO before steal for no-home workers
Workers without a replica-host match take leftover first; steal (K/orphan)
only when FIFO is empty. Home workers keep own-queue, steal, then leftover.
---------
Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
2026-10-13 05:31:34 +08:00
Bruce ZouGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: add conditional grow by average fill ratio to VolumeGrow gRPC
Callers that pre-grow volumes on fill level currently have to list all
volumes of a collection and sum sizes themselves, which is expensive on
clusters with millions of volumes. The master already tracks each
writable volume's effective size in memory, so it can answer the
question directly.
Add grow_avg_fill_threshold to VolumeGrowRequest: when set (e.g. 0.6),
growth only happens once the layout's writable volumes have reached
that average fill ratio. A layout with no writable volumes skips the
check, since that is the case growth exists to fix. VolumeGrowResponse
now reports the observed avg_fill and writable_count whether or not
growth was triggered.
* master: unit tests for WritableAvgFill and FindVolumeLayout
Covers the average-fill computation (mixed fills, no writables, a
writable volume without a size-tracking entry yet) and that
FindVolumeLayout resolves existing layouts without creating missing
collections.
* address review: report avg_fill and writable_count for every VolumeGrow
The stats were only read when grow_avg_fill_threshold was set, so a
request without a threshold got AvgFill=0 and WritableCount=0 even when
writable volumes existed. Read the layout stats unconditionally and use
the threshold only to decide whether growth happens.
* master: scope the grow fill check to the requested target
A VolumeGrow aimed at a data center, rack or node must not skip growth
because fill elsewhere keeps the layout average below the threshold.
Measure only writables serving the target; a target with none still
grows.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 05:30:03 +08:00
Jay HemnaniGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sftp: fail a directory listing on a stream error, and page past hidden entries
* sftp: test a fully hidden listing page
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 19:34:46 +08:00
Alex KGitHubChris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: copy a remote-only entry on the remote when renaming it
Renaming a file under a remote.mount whose content exists only on the
remote (never cached, or remote.uncache'd) lost it: the new entry kept
the old RemoteEntry, deleting the old entry deleted the old object, and
with no chunks nothing ever wrote the new key. filer.remote.sync could
not finish the rename either, since the old object was already gone.
moveSelfEntry now copies such an entry's object to the new key before
it touches anything, and the new entry gets the RemoteEntry of the
copy. The copy is a new optional RemoteStorageObjectCopier interface,
implemented for S3 with CopyObject up to 5 GB and UploadPartCopy above
that; the multipart copy carries the source's metadata and, where the
remote supports tagging, its tags. When an existing target is
overwritten, its delete keeps the remote object (the copy replaced it)
and its event is marked metadata-only so the sync leaves it alone too.
A remote-only rename that leaves its mount, or whose remote cannot
copy, is refused instead of deleting the only copy. Empty entries and
entries with local data rename as before.
UpdateFileMetadata builds its copy source with the same helper, which
escapes each key segment and keeps the slashes.
* filer: delete a copied rename's old object only after the commit
Review follow-ups:
- The old key of a copied remote-only entry, and the old directory of a
rename that copied one, are now deleted once the rename has committed.
A rename that fails after the copy leaves the source entry and its
object in place, plus an extra object at the new key.
- Every part of a multipart copy is conditioned on the ETag the copy
started from, so an object overwritten mid-copy fails the copy
instead of mixing versions.
- Both copy paths keep the source's server-side encryption settings.
- filer.remote.sync no longer deletes a key that a remote-only entry
holds by the time the delete event is processed: a file deleted and
then replaced by a remote-only rename has its content only there.
* filer: do not sweep the old prefix of a directory rename that copied
Removing the old directory on the remote deletes everything under its
prefix. Run after the commit, that could also take an object another
rename copied in under the same path meanwhile. When a child was copied,
leave the prefix alone and delete only the copied children's old keys.
* filer: keep an old key a remote-only entry has retaken by cleanup time
A directory rename does not lock its children's paths, so between the
commit and the deferred delete of a copied child's old key another
rename can put a remote-only entry there, whose content is then only in
that object. Check the store before deleting.
* filer: keep a renamed entry's old object when the path lookup fails
The deferred delete must only run once the lookup shows no remote-only
entry at the old path. A failed lookup shows nothing, and an extra
object loses nothing.
* filer: copy over a remote-only rename target only after the commit
Copying the source object onto the target's key before the transaction
committed destroyed the target's only content, which a rolled-back
commit cannot restore. When the target is remote-only the copy now runs
after the commit: the new entry keeps the old key's stamp until the copy
lands and re-stamps it, the target's object is kept meanwhile, and a
failed copy keeps the old object so remote.sync can finish it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote.sync: match the delete event's etag before deleting the object
A rename can copy another entry's content onto a key between the filer's
check and the remote delete, and the delete then removes the copy. The
delete event carries the object's etag; delete only when the live object
still matches it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: finish a rename's remote work outside the committed transaction
The SQL stores keep the transaction in the context and refuse every
call through it once it has committed. The work that runs after
CommitTransaction used that context, so on postgres2 the deferred
old-key delete could not look up the old path and kept the object, and
finishRemoteCopies could not read or stamp the entry. Both failed with
"sql: transaction has already been committed or rolled back" on every
copied rename; remote.sync cleaned up after them.
Keep the context from before the transaction for the work after the
commit. The rename test store gets a strict mode that fails calls
through a committed transaction the way database/sql does.
* filer: carry a deferred copy's stamp in the rename event; delete orphaned uploads
- The deferred copy stamps the entry with a plain UpdateEntry, which
publishes no event, so the queued rename event still carried the old
key's stamp to peer filers and subscribers. Give it the copy's
RemoteEntry before it is sent. The two notify loops share the code now.
- remote.sync kept any object whose ETag changed since the delete. With
no entry at the path that is an upload which finished after the
delete, and keeping it leaves it on the remote for good. Look up the
path first and keep a changed object only when an entry holds it.
* remote.sync: keep a changed object even with no entry at its path
Deleting a changed object when no entry holds the path also deletes a
copy a directory rename has written but not yet made visible, while a
delete of that path waits in the sync; the rename then removes the old
key and the content is gone. Keep any object whose ETag changed since
the delete, as before. An upload that finishes after the delete leaks
its object instead, which is the cheaper failure.
* filer: read a pending remote rename copy from the source key
A remote-only rename over a remote-only target commits before its copy
lands, so reads on the new path served the target's old object until the
copy stamped the entry — and indefinitely when the copy failed without
remote.sync running. The committed entry now records the renamed-from
path, and remote-only reads resolve that object's key while the stamp
is pending; the copy's stamp and the sync's completion stamp both clear
the marker.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: stamp the rename event with the deferred copy's result
A remote-only rename over a remote-only target queued the new entry's
event with the source's stamp before the deferred copy ran. Peers
replaying it saved a stamp the copy had already superseded, and the
sync serving such a peer rejected the destination as a match and tried
to copy the deleted source. The event now carries the copy's result
and drops the pending-source marker once it lands.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote.sync: delete only the object the event's stamp describes
An object at a deleted key is not always the one the delete meant: a
rename copy can land before its filer entry is visible, and a rewrite
can upload over it. When the event recorded no remote stamp, or the
live stat cannot confirm the recorded etag, the delete now keeps the
object instead of treating it as a late upload.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: tighten the pending rename source marker
Follow-ups to reading a pending remote rename copy from its source key:
- Only the filer sets the marker. The HTTP upload and tagging handlers
store Seaweed-* request headers as extended attributes, so they skip
this one, and a marker that leads out of the entry's own mount is
ignored, so it cannot point reads at another remote.
- A pending entry renamed again copies from the marker's source, not
from its own key, which still holds the old target's bytes. A further
deferred rename keeps naming the original source; an immediate copy
clears the marker.
- When remote.sync finds the destination already holding the content,
it stamps the entry (clearing the marker) before deleting the source,
and deletes the source only once the entry no longer reads it.
- A read that resolved the source just before the copy finished and
the source went away retries with the entry as it is now.
* filer, remote.sync: follow a pending rename's source across renames
When a remote-only entry is renamed over another remote-only file and
the deferred copy fails, it reads the source key through its marker.
Renamed again the same way, the marker keeps naming that source, but:
- remote.sync completed each rename from the event's old key. For the
second one that is the first target's old object, which it copied
over the destination before clearing the marker. It now copies from
the source the entry's marker names.
- For the first rename it found no entry at the new path and deleted
the old key, which the later entry still reads. When the event's new
entry carried a marker and the path is gone, it now leaves the key to
the entry that moved on.
- When the filer copies a pending entry immediately on a later rename,
it now also releases the source the copy read, after the commit and
with the same checks as the old key.
* remote.sync: release a pending rename's source, the filer leaves it
The filer deleted a pending entry's source after copying it on a later
rename, but that rename locks only its own paths: another rename can
copy onto the source key before its entry is visible, and the cleanup
then deleted that copy. And deleting a pending entry never released its
source at all.
The filer now leaves marked sources alone. remote.sync releases one
when the pending entry is deleted, or when a later rename was copied
right away, and only while the live object is still the one the
entry's stamp describes and no remote-only entry holds the path: a
copy that landed there since has a different ETag and is kept.
* remote.sync: release a pending source when the renamed entry is cached
A rename copied right away left its pending source to remote.sync,
which released it only while the destination was still remote-only.
Cached before the sync got there, the destination took the other path
and the source stayed on the remote for good. Release it there too,
with the same checks.
* remote.sync: release a pending source only after the destination is safe
e6d6438 released the source a renamed entry's old version read before
the destination was uploaded. A cached destination that still carried
the pending marker then had nothing behind it: had its upload failed,
remote.uncache could drop its chunks and leave it reading a deleted
source. Release the source only after the upload is stamped, and only
when the entry no longer reads through a marker.
* remote.sync: delete a rename's old key only if it is still the renamed content
completeRemoteOnlyRename deleted the old key once the destination held
the content. Another rename onto the vacated path can copy its object
there before the sync reaches the first rename, and that copy was then
deleted with it. Delete the old key only while the live object is still
the one the renamed entry's stamp describes.
Also keep the pending-source marker out of DELETE ?tagging, which
clears every Seaweed-* attribute: the marker is not a tag, and losing it
points reads back at the old target's bytes.
* remote.sync: compare a rename's old key against the source's own stamp
A retry can find the renamed entry already stamped with the copy, and a
copy of a multipart source has another ETag, so the unchanged source
was kept and stayed on the remote. Carry the stamp the event recorded
for the source (or the pending entry's, for a marked source) and compare
against that.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 19:21:54 +08:00
ingardmGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Claude Opus 5.5Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: extract the periodic per-layout grow decision
Move the must-grow and crowded-step decision of the periodic growth scan
in ProcessGrowRequest into periodicGrowCount, a pure function of the
layout's lastGrowCount, writable and crowded counts, so it can be tested
without running the scan loop or growing real volumes.
No behavior change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: record writableVolumeCount on HTTP /dir/assign
The periodic growth scan (#5781) keeps a layout supplied ahead of
demand only when it knows how many writable volumes clients want:
lastGrowCount, which gRPC Assign records with SetLastGrowCount. HTTP
/dir/assign parsed writableVolumeCount but only used it to size the
emergency grow when no writable volume was left, so the scan always saw
lastGrowCount 0 for HTTP-only writers.
Since #5821 removed the per-assign threshold grow, such a layout only
grew by the rack-aware step of 2, and only once every writable volume
was crowded. When a batch of volumes filled in lockstep, those 2 took
every write and could fill before the next scan, leaving no writable
volumes until an assign-triggered emergency grow landed, with 503s for
every other assign meanwhile.
Record writableVolumeCount on HTTP assigns as gRPC Assign does. Clients
that do not send it are unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: reject unknown rack/dataNode before recording grow targets
An assign naming a rack or data node outside the topology can never be
served, but lastGrowCount was recorded first. The periodic growth scan
drops the requested placement, so it kept topping the layout up with
volumes the requester could not use. Both assign paths now validate the
placement like they already validate the data center.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: scope placement existence checks to the requested path
A rack in another data center or a data node under another rack passed
validation yet could never serve the request, and the grow count was
still recorded. The rack and data-node lookups now take the requested
parents, matching the filters the placement engine applies.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: rotate EC extra shards on pickTarget count ties
pickTarget used to keep the first sorted key when shard counts tied,
so 10+4 onto 4 racks always put the two fours on full-01/full-02.
Tie-break on remaining freeSlots (same idea as pickNodeInRackEligible)
so the leftover shards walk across racks over a series of volumes.
* fix: pay within-rack EC shard debt on Place
Score nodes by live EcShards plus this Place's assignments, not
this-volume counts or EC-volume presence. Allow SameRackCount+1
only after a sibling already holds a shard, so 4-on-2 becomes 3+1
while 3-on-2 stays 2+1. Rack pickTarget tie-break is unchanged.
* fix: even-spread EC disks, pay debt with remainders only
Do not pack 3+1: three shards on one disk is one disk-loss from
unrecoverable. Place this volume evenly first; leftover shards
(2+1) go to the thinner live count. 4-on-2 stays 2+2.
* fix: score Place extras by freeSlots then liveShards
Even-per-volume and SameRackCount stay; remainder prefers capacity,
then shard debt. No 3+1.
* fix: keep shard-debt fixture capacity-consistent
freeSlots = cap-shards so extras keep paying the thinner disk
across volumes instead of oscillating after the first pair.
* refactor: document dropped mFree; switch EcBalance node pick
Keep host even (mCount). Remainder is disk freeSlots then liveShards so
thin disks fill first. SameRackCount / 2+2 unchanged. pickTarget unchanged.
* test: cover Place 10+4 extras on thin disks/racks
Seed liveShards/freeSlots imbalance on 1-1-{4,6,7,8,15}, 4-2-3, 5-2-2, 6-1-2, 7-1-2. Remainder tails must miss fat first ids; remainder 0 freezes the live gap.
* fix(ecbalancer): count only matching-type slots when picking a move target
Aggregate freeSlots let a rack, machine, or node whose free capacity sat
on another disk type win placement; pickBestDiskOnNode then found no
eligible disk and the move went out with TargetDisk 0, which the copy
endpoint reads as automatic selection onto any disk.
Score candidates on freeSlotsForType — per-disk metadata when present,
the node aggregate when absent — so a wrong-type destination is never
chosen, and cover pickBestNodeForVolume's tie-breaks and the wrong-type
plan case.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec_balance: spread free slots per disk type, keeping the remainder
Dividing the node's free slots evenly over all its non-full disks floored
a leftover slot to zero on every disk — one free slot across two disks
made the node look full. It also gave disks of other types a share of
slots counted only for the requested type. Each type's slots now spread
over that type's disks with ceil division, so the total is preserved and
every usable slot stays visible to the planner.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:09:53 +08:00
Joe PassananteGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(volume): do not read a needle's data to delete it
A delete read the needle's whole payload from disk only to discard it.
The delete handler and BatchDelete need the needle's cookie and its
chunk-manifest flag, which live in the header and trailer, but fetched
them with a full needle read, the same call a GET uses.
readNeedle now reads the header and trailer only when MustMetaOnly is
set, whatever the needle's size, and the delete paths ask for that. The
Rust volume server honours must_meta_only the same way for its delete
paths. A compressed needle or a chunk manifest still gets the full read,
as before, because their callers need the bytes.
HEAD is unchanged. It used to set MustMetaOnly, which nothing read; it
now asks for AttemptMetaOnly alone, so it keeps reading a needle up to
PagedReadLimit whole. On a spinning disk that one read is cheaper than a
header read followed by a trailer read.
Version 1 needles are always read in full. They have no data size field
and no metadata after the data, which ReadNeedleMeta relies on, so a
metadata-only read of one failed with "invalid needle meta size". That
already happened for version 1 needles above PagedReadLimit read with
AttemptMetaOnly.
A delete therefore no longer verifies the data CRC of the needle it
removes. The cookie check is unchanged.
* fix(volume): read transformed images fully on HEAD too
A HEAD carrying resize or crop params reports the transformed length,
which a metadata read cannot produce. Go and Rust both answered such
HEADs from the source header, returning the stored size instead of the
cropped or resized one.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The admin control stream now requires mTLS or a token signed by
jwt.filer_signing.key; the Rust worker gets --filer-signing-key and
--filer-signing-key-expires-after-seconds, mirroring the Go worker's
config names, and attaches the token to the stream metadata.
Workers mint a SeaweedFilerAdminClaims token when jwt.filer_signing.key
is in their configuration, matching the authentication the admin server
now requires.
Worker registration previously ran unauthenticated on any interface the
listener bound. The stream now requires a verified mTLS client
certificate or a Bearer token signed by jwt.filer_signing.key, and falls
back to local clients only when no key is configured.
The local-peer fallback and bearer-token validation used by the filer IAM
gRPC path move into weed/security so every admin-plane gRPC service can
enforce the same contract. A client certificate verified by the server's
TLS config now also counts as authenticated.
A path whose ancestors are all absent recursed on the root forever,
since Dir("/") is "/".
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
newFileWriter authorized uploads against the directory, so a file with
a more specific read-only ACL or mode could still be overwritten. The
same parent-directory fallback used for nonexistent paths now applies
through CheckFilePermission on the target itself.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CreateMultipartUpload accepts canned and custom ACLs on initiation;
run them through the same authorization path as PutObject so an
explicit ACL requires s3:PutObjectAcl, and persist the validated
grants on the upload entry for completion to apply.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Multipart initiation stored client-supplied Seaweed-X-Amz-Acl and
Seaweed-X-Amz-Owner request headers onto the upload entry, and
CompleteMultipartUpload copied them to the finished object, letting a
caller forge ownership or a public ACL without s3:PutObjectAcl.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The decoupled-delete guard only inspects the resolved data path itself,
but the catalog marker lives on a table's name path, so a location that
resolves inside another table's directory (its marker-less data
subdirectory) slips through and the recursive purge destroys the other
table's data. Walk the data path's ancestors within the bucket and
refuse when any carries a table or view catalog entry.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CreateTable/RegisterTable/UpdateTable (and the view equivalents) validate
the location only for bucket and traversal, so a tenant can register a
table whose location resolves into another namespace's directory and
DeleteTable will purge it. Require the location to live under the
caller's namespace subtree, the boundary per-namespace delegation uses.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
ChunkReadAt.ReadAt asks the ReaderCache to prefetch after every read,
and MaybeCache counted only the downloads it newly started toward the
window, so each call reached prefetchCount chunks further than the last.
A reader consuming a chunk in several reads, as the S3 GET path does
with a 256 KiB copy buffer over 4 MiB chunks, ran the read-ahead up to
the ReaderCache downloader limit and kept most of a large object in
memory, enough to get an S3 gateway OOM-killed mid-response.
Count every chunk in the window, downloading or not, so the read-ahead
stays at prefetchCount chunks past the reader.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-12 17:41:01 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: skip replayed peer events older than the local entry
Replay deleted and re-inserted every received event unconditionally, so
a restarted filer that rewound its subscription offset re-applied stale
peer writes over newer local state (values reverted, deleted entries
came back and pointed at already-vacuumed chunks). Compare the event's
TsNs against the local entry's Mtime and skip per-path, so a move event
can still remove its source while keeping a newer destination.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: add replay last-writer-wins regression tests
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: stop peer subscriptions and flush offsets on shutdown
Filer.Shutdown closed the store while subscription goroutines were still
mid-stream; their deferred offset save was never guaranteed to run, so a
restart rewound each peer's stream back to the last periodic flush and
replayed stale events. MetaAggregator.Shutdown now closes every peer's
stop channel and waits for the loops so the saved offset covers what was
replayed, and the reconnect sleep is interruptible by the stop channel.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: test that shutdown stops and waits for peer loops
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: order replay by persisted per-path commit timestamps
Review feedback found the mtime comparison still lets stale replay win:
a local delete removes the entry it would have been compared against, an
entry mtime is client-controllable, and the check ran unlocked so a local
write could land between it and the store mutation.
- Record the event timestamp per mutated path in the store KV on every
emitted mutation, plus a delete record for sources of deletes and
renames. The record survives the entry, so a replayed create older than
a local delete no longer resurrects it; ancestor delete records cover
children swept without per-child events.
- Replay compares the event timestamp against that record under the same
per-path entry locks local RPC mutations hold, falling back to entry
mtime only for entries written before records existed. Lookup errors
now abort the event instead of being treated as an absent entry.
- MetaAggregator.Shutdown marks the aggregator stopped under peerChansLock
so a master update cannot start a subscription that outlives the wait,
and waits for the loops without the five-second cap now that every
blocking stage — signature RPC, bootstrap traversal, retry sleeps, and
per-event processing — cancels on the stop channel.
* filer: pin peer offset at failed replay, fix commit-record context and key size
A replayed event that exhausted its retries (or was interrupted by
shutdown) was still followed by advancing the peer watermark, so the
persisted offset skipped the unapplied event forever. replicateMetadataChange
now reports the outcome: errPeerSubscriptionStopped aborts the subscription
so the deferred save keeps the last completed event, and an exhausted retry
records a barrier the persisted offset and reconnect resume point cannot
pass while the stream keeps flowing.
Commit records also failed on two fronts: NotifyUpdateEvent runs after
CommitTransaction with the same context, so SQL stores wrote MpTs/MpDl
through an already-committed *sql.Tx; and metaPathKey embedded the whole
path, overflowing the 766-char name column SQL KV stores use, which the
KvPut fallback update then silently dropped. The writes now use a fresh
context and a sha256-hashed fixed-size key.
* filer: lock HTTP delete handler and replay against ancestor paths
The HTTP DeleteHandler called DeleteEntryMetaAndData without the shared
entry lock, so a peer replay could still interleave with a local delete and
its commit record. It now holds the same lock the gRPC DeleteEntry handler
does.
Replay also locked only the exact event paths, but its ordering check
consults ancestor delete records: a local recursive delete holds only the
ancestor's lock, so a stale child create could check before the tombstone
landed and then reinsert. lockReplayPaths now locks every ancestor of both
event paths in one lexicographic order — parents sort before children, as
the rename handler's two-path order already does — while never locking the
root, which cannot be deleted and would serialize all replays.
* filer: record replay commit before the replacement insert
If the same-path update DeleteEntry succeeded but InsertEntry failed, the path sat deleted with no commit record until the retry arrived, and a stale peer create could resurrect it in the gap. The commit record now lands immediately after the delete — without a delete record, since an update is not a delete — and again after the successful insert.
* filer: lock replay ancestors shared, not exclusive
Exclusive locks on every ancestor serialized all replays under a shared
parent — unrelated objects in the same bucket could no longer apply in
parallel. Ancestors only need to exclude an exclusive holder (a local
delete or rename of the ancestor), so they now take shared locks; a path
the event changes stays exclusive even when it is also the other path's
ancestor.
* filer: order peer bootstrap on the sender's commit timestamp
traversePeerMetadata compared localCommitTsNs — a commit-clock value —
against the entry's mtime, which a client can set freely: a peer change
committed recently with an old mtime was skipped, and a stale entry
mtime-stamped in the future overwrote a newer local commit.
TraverseBfsMetadataResponse now carries the sender's per-path commit
timestamp (recorded MpTs record, entry mtime as the unrecorded fallback).
The bootstrap applies an entry only when its commit is newer than the
local one and records the applied timestamp, so replayed events order
against it — closing the gap between bootstrap content and the event
stream. A peer without the field sends 0 and keeps the mtime fallback,
preserving rolling-upgrade behavior.
* filer: keep failed commit records pending until they land
notePathChange ran after the mutation committed and only logged a KvPut
failure: a delete whose MpTs/MpDl record never landed left no tombstone,
so a stale replayed create could resurrect the path while the store was
unreachable.
Failed records now stay in metaTsPending and a repair pass retries them
until they land. localCommitTsNs, deletedAfterTsNs and CommitTsNs
consult the pending record, so ordering holds in the window between the
committed mutation and the durable record. A per-path flag bounds the
repair to one goroutine, and respawns cover merges landing between its
last pass and the flag release.
* filer: keep only unwritten work in the pending commit record
The pending record merged every obligation into one timestamp and flag,
and a later successful write left it untouched: a repair pass could then
write an older timestamp over a newer one, and a merged delete inherited
the newer change's time, which the ancestor check reads as a newer
delete. The record now holds a change and a delete timestamp of its own,
both discharged under the record's lock alongside the store writes that
cover them, so a repair writes only what is still owed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: fail traversal on a commit-record read error
CommitTsNs answered a store read failure with the entry mtime, a value
clients can set arbitrarily far ahead; a peer bootstrapping on it would
record the minted time and reject real updates below it. The lookup now
returns the error so the traversal restarts instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 17:39:34 +08:00
Ege ZambelliGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Ege ZambelliChris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* helm: keep the S3 config checksum stable across the first upgrade (fixes#11719)
checksum/s3config hashed the rendered s3-secret.yaml. That template
keeps its generated S3 keys with lookup, so rendering it again for the
checksum drew new keys whenever lookup found nothing: on install the
filer, S3 and all-in-one pods each carried a checksum of a different
set of keys than the Secret got, and the first upgrade with unchanged
values corrected it and restarted them.
Hash the values s3-secret.yaml is built from instead: the enable and
auth switches, existingConfigSecret, credentials, the legacy-secret
settings, the audit log configs and the Secret name. Only those, so
scaling the S3 gateway or changing its resources does not restart the
filer. A change to any of them still restarts the pods. Keys that the
chart generated stay out of the hash; they only change if the Secret
is deleted by hand.
The new render test fails on the old template (two checksums for one
Secret, different on every render) and passes now; it also checks that
the checksum follows the credentials and filer.s3.auditLogConfig and
ignores s3.replicas and s3.resources. On kind, an upgrade with unchanged
values no longer replaces the filer pod.
* helm: check the S3 checksum per workload, all-in-one included
Each workload's checksum/s3config is now read on its own (--show-only),
so a workload that loses the annotation fails the test instead of being
hidden by sort -u, and the all-in-one deployment gets the same checks.
Verified against two broken copies: one without the filer annotation,
one with the old all-in-one checksum; the test fails on both.
The helper comment now says what happens when the Secret is deleted to
rotate generated keys.
* ci(helm): fail the S3 checksum test on a missing or unrendered checksum
A failed helm template or a render without the annotation produced an
empty checksum, and the inequality assertions then passed against it.
s3_checksum now returns non-zero unless it found a checksum, and the
changed-value renders are assigned before comparing so failures cannot
hide inside test.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Ege Zambelli <egezamb@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test/fuse_dlm, test/fuse_p2p/peer_chunk_sharing_test.go and
unmaintained/check_disk_size use syscall.Flock, syscall.Stat_t and
syscall.Statfs, which do not exist on Windows. Without constraints,
gopls and go vet on a Windows host report them as undefined.
fuse_p2p/framework_test.go already carries `linux || darwin`; the FUSE
test files now match it. check_disk_size uses the same constraint as
weed/stats/disk_supported.go.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-12 09:53:24 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: propagate request context through metadata lookups
The FUSE interrupt channel never reached the filer RPC: lookupEntry ran
its LookupEntry under context.Background(), so a stalled filer parked
ops like ListXAttr forever even after the kernel gave up on them.
Thread a context from each op's cancel channel through
maybeReadEntry, maybeLoadEntry, lookupEntry, ensureDirectoryVisited and
waitForPendingAsyncFlush; the existing withFilerClient attempt bound
turns it into both an interrupt and a deadline. A caller cancellation
now surfaces as EINTR instead of waiting out the RPC.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: reuse ctxFromCancel for the read path's interrupt bridge
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: regression tests for canceled and stalled filer lookups
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: thread ctx through AcquireHandle
Open, Create and Mknod all carry the request's cancel channel; let it
bound the pending-flush wait and metadata load inside AcquireHandle too.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: keep flush waits and write sequences uninterruptible
The cancel-derived context must only cover metadata reads: an
interrupted Unlink or Rename that skips the pending-flush wait can
race the async write back into existence, and Link/Rename mutations
that run under a canceled context abort mid-sequence, strand the
rollback, and leave wrong hard-link counts. Restore the blocking flush
wait and run each op's write phase on context.Background(). Rmdir and
Unlink also propagate EINTR from the parent lookup instead of
proceeding past a permission check the interrupt prevented.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: interrupt shared directory builds and directory reads
Waiters behind another request's singleflight directory build blocked
until it finished even when interrupted; switch the wait to DoChan so
the request context can abandon it. ReadDir paths now carry the
request context through EnsureListingFresh, cache listings and direct
reads, and map its cancellation to EINTR rather than EIO.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: assert exact statuses in filer-stall regression tests
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: let read-side flush waits release on interrupt
waitForPendingAsyncFlush reports whether the flush actually finished:
lookupEntry and AcquireHandle return EINTR when the request is
abandoned mid-wait, while Unlink and Rename keep waiting on
context.Background() so the delete/rename still sees the flushed
state.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: propagate interrupt through remaining lookup checks
Rename's source and destination permission lookups now return EINTR
on interrupt instead of skipping checkStickyBit or masking an
existing target, and the readdir preload visit maps cancellation to
EINTR like the other branches.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: fail closed on errored rename prerequisite lookups
* mount: stop rmdir on interrupted target lookup, fail closed on dir lookup errors
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 09:42:52 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: commit v3 row lineage on the three commit paths
iceberg-go 0.7.0 refuses every snapshot on a format-version 3 table
that lacks first-row-id and added-rows ("invalid row lineage"), so no
v3 table was ever compacted or had its manifests rewritten; worse,
passing 0 to the manifest list would have handed new files row IDs
existing rows already hold.
Each commit now plans first-row-id from the table's next-row-id, lets
the v3 manifest-list writer assign manifest first_row_id from there,
and reports the assigned range as the snapshot's added-rows so
next-row-id advances by exactly that. The commit guard also treats a
moved next-row-id or a changed format version as a stale plan, like a
moved head: a commit to another branch advances next-row-id too, and
manifests written for the old version must not be committed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: keep row identity through v3 compaction
A rewrite that moves rows into a new data file must not renumber them.
On v3 tables the merge now appends _row_id and
_last_updated_sequence_number (spec field ids 2147483540/2147483539)
to the output and fills each row with the value its input
materializes, else the inherited one: the input file's first_row_id
plus the row's position before any delete, and the manifest entry's
data sequence number. Inputs are matched with their lineage columns
set aside, so a file an earlier compaction wrote merges with one an
engine wrote; a duplicate lineage column or a reserved name under
another field id fails the bin instead of writing a doubled column.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: skip rewrite_position_delete_files on v3 tables
The spec forbids adding position-delete files to a format-version 3
table (v3 writes deletion vectors), and this rewrite adds new ones.
Detection no longer proposes the operation and the operation returns a
skip result; compaction still applies and removes existing position
deletes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: keep Puffin deletion vectors out of position-delete reads
A Puffin file in a delete manifest is a deletion vector, not a
position-delete file: reading it as Parquet fails, and merging the data
file it covers without applying the vector resurrects deleted rows.
Skip Puffin entries in the position-delete collection and the delete
rewrite group scan, and exclude the files a vector covers from
compaction candidates - in compactDataFiles and in eligibility
detection alike. A vector whose referenced_data_file is missing or
unresolvable fails the run rather than silently compacting whatever it
points at.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: name a merged file's own first row ID
Rewritten rows keep their IDs, so a merged data file's first_row_id is
the _row_id of its first written row - not the next free range the
manifest list would otherwise assign to the entry at commit. Read the
first materialized value back off the merged output and stamp it on the
manifest entry.
Add an end-to-end test that populates a v3 table whose entries carry no
first_row_id, so the compact path exercises manifest first_row_id
inheritance through the stored manifest list, then commits and reads
the result back through the real manifest readers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: locate the _row_id column by index, not row position
Repeated columns shift parquet.Row positions, so leaf.ColumnIndex is not
the value's slot — a first row containing a list or map saved an unrelated
value as the merged file's first_row_id.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: read the returned row before treating EOF as empty
parquet-go can return a row alongside io.EOF; a one-row merged file then
lost its first_row_id and the manifest list would allocate a fresh range.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: avoid nil dereference in first-row-id failure message
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: resolve same-second updates with the sync stamp
shouldSendToRemote compared remote_mtime to the entry mtime in whole
seconds, so a rewrite landing in the same second as the previous upload
was dropped and never retried. When the two seconds are equal, fall back
to the full-precision LastLocalSyncTsNs stamp, the same check
remote.uncache uses to refuse uncaching an unsynced entry. An entry
without the stamp is treated as unsynced and sent.
Fixes#11713
* filer.remote.sync: restamp the entry after a metadata-only remote update
A same-second metadata edit now reaches UpdateFileMetadata, but without
advancing LastLocalSyncTsNs the entry stays dirty and every later event
re-sends the metadata update. Stamping it through updateLocalEntry keeps
the same IF_ENTRY_EQUAL guard as an upload stamp.
* filer.remote.sync: stamp a cloned RemoteEntry in updateLocalEntry
Callers can pass the event's own RemoteEntry; mutating its stamp before
the expected entry is cloned put the new stamp into IF_ENTRY_EQUAL while
the filer still holds the old one, so every restamp was rejected as
stale. Stamping a clone keeps the precondition on the stored stamp and
leaves the caller's RemoteEntry untouched.
* filer: prevent nil pointer dereference in CompactFileChunks when context is cancelled
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* filer: propagate chunk resolution errors through ViewFromChunks
A failed manifest resolution produced a nil interval list; with the nil
guards this silently became an empty view, so replication sinks would
write empty replicas and report success. Return the error so callers
fail instead, and make ChunkStreamReader report it on first read.
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* wdclient: gate master registration until the server is serving
The first KeepConnected send registers the client into the master's
cluster list, which is what routes S3 and peer traffic to it. A filer
connects to the master long before its gRPC port serves (peer discovery,
store bootstrap), so it announced itself while still refusing
connections. announceCh defers only the registration send, letting the
stream machinery connect early without joining the list; ListMasters
exposes the configured masters for callers that cannot wait on the
stream.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: announce to the master only once gRPC serves
NewFilerServer previously registered on the master's filer list during
startup, before the gRPC listener was serving — and on a fresh store
before peer bootstrap finished — so a restarting filer could be routed
to while refusing connections. The announce channel is closed right
after grpcS.Serve starts. ListExistingPeerUpdates now iterates the
configured masters directly since GetMaster would block on the gated
stream.
On shutdown, cancelling the master context ends the KeepConnected
stream during the leave phase, so the master drops the filer node
before gRPC stops accepting; fs.Shutdown cancels it as a fallback.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* wdclient: test that the announce gate delays registration
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* wdclient: unblock queries behind the announce gate
The gate holds the registration send, but setting currentMaster only after
the first response left GetMaster blocked for the whole gated window. Any
startup path that queries the master before serving (e.g. loading a
chunked filer.conf) deadlocked the constructor, so the gate never opened.
Mark the master queryable once the stream is established; registration and
leader redirect still wait on the gate and first response as before.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep the fullest peer list from any master
ListExistingPeerUpdates returned the first non-nil answer, so a follower
master could seed the peer list with a stale partial view before the
leader replied. Keep the longest response instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep master queries usable after the shutdown disconnect
Leaving the filer list early ends the KeepConnected stream, which cleared
currentMaster and left the final metadata-log flush unable to reach the
master during drain. Keep the last serving address queryable after a
canceled disconnect, and move the Shutdown fallback cancel behind the
filer's own shutdown so callers that skip LeaveLockRing flush against a
live stream.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* cluster: propagate a bounded context through ListExistingPeerUpdates
The helper dialed and queried masters on context.Background, so one
unresponsive master could stall the whole peer scan and ignore caller
cancellation. Pass ctx through, bound each master query in the filer's
scan, and stop when the caller cancels.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: wait for cluster removal before draining gRPC
Cancelling the KeepConnected stream only signals removal; the master
drops the filer when its stream handler unwinds. Until then new requests
can still route to this filer while its listener is closing. Poll the
configured masters, bounded, until none still lists this filer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* wdclient: serialize master discovery access
ListMasters made peer scans a second concurrent accessor of mc.masters
alongside tryAllMasters, while ServerDiscovery refreshes and reads its
address list unsynchronized. Guard the refresh and every instance copy
behind one lock.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* cluster, filer: prefer the leader's peer list and report lookup errors
Keeping the longest answer let a stale ex-leader's list win: after a
leader change the old master retains registrations until its streams
end, so a fresh filer could bootstrap from an unreachable peer. Ask the
masters for the raft leader and take its list; only when the leader is
unreachable fall back to the fullest answer.
ListExistingPeerUpdates also returned an empty slice on RPC failure,
which a fresh filer read as "no peers" and skipped bootstrap, serving
without the existing files. Propagate the error and retry the scan
during startup, before the announce gate opens.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound the removal wait and retry unconfirmed masters
Each master query got its own two-second timeout while the five-second
deadline was only checked between scans, so three stalled queries could
stretch the wait past its budget. A failed query also counted as
confirmed removal, letting gRPC drain while a master still listed the
filer. Derive every query context from one overall deadline, track which
masters confirmed absence, and retry the rest until they answer or the
budget expires.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* wdclient: run the SRV lookup outside the masters lock
ListMasters and tryAllMasters held mastersLock across the DNS lookup, so
a slow resolver stalled GetMasters and ListMasterSet readers past their
request deadlines. Split the lookup from the list swap: resolve the SRV
record unlocked, then lock only to save it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* wdclient: bound WaitUntilConnected in the announce-gate test
The wait ran on the caller context with no deadline, so a regression
that left currentMaster unset hung the whole test run instead of
failing the check. Give it the same three-second budget as the
assertions that follow.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* wdclient: order SRV refreshes and reuse the cached list on shutdown
* filer: confirm an empty leader peer list before accepting it
* filer: announce only after the startup peers' replay catches up
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: bound each filer call so a wedged filer cannot hold the failover walk
WithFilerClient callers run their RPCs on context.Background(), so a filer
whose transport stalls without refusing (stopped peer, black-holed
connection) blocks the call forever and the walk never reaches the
remaining filer addresses. Run each attempt under a timeout; on expiry
drop the cached channel so the abandoned call is unblocked and the next
address is tried.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: time out wedged mutation streams and rotate to the next filer
doUnary and Rename wait on stream Send/Recv under context.Background(); a
dead transport never delivers an error, so writes hang without ever
reaching the unary fallback. Bound the waits and tear the stream down on
timeout; teardown also advances the sticky filer index so the next stream
opens against a different filer instead of redialing the dead one.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: cover filer failover past a wedged filer
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: cancel filer attempts by context and unwind them before retry
Review follow-ups on the filer failover:
- callFiler derives a per-attempt context instead of only abandoning the
wait, so a timed-out attempt's RPCs die on their own without tearing
down the shared cached channel. The walk now also waits for the
attempt's callback to unwind before trying the next filer, so two runs
of the same closure cannot race on the caller's captured results.
Callbacks that ignore the attempt context are still forced out by
dropping the channel, as a last resort after a grace period.
- teardownStream captures the dying stream's filer index and rotates it
while holding m.mu, so it can never rotate a successor generation's
filer by mistake.
- The unary rename fallback bounds silence on its dedicated stream the
way the mux does, instead of trusting Recv on a wedged filer.
- downloadRemoteEntry opts out of the metadata-call bound: a healthy
whole-file download can legitimately outlast it.
* mount: count only wire silence against the rename timeout
Applying a rename event can outlast the silence bound on a healthy
filer — a directory rename migrates an open child's DLM lock under
another writer. Stop the mux's response timer for the local work and
re-arm it before the next wait; the fallback stream's watchdog defers
while the handler runs instead of cancelling a live stream.
* mount: restart the rename fallback's silence bound after each event
The bound covers the wait for the next response, not the rename's whole
lifetime — a healthy rename that keeps sending events can outlast it.
Re-arm after each handled event; the defer-while-handling stays for
local work that outlasts the interval on its own.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: preserve remote object ETags across cache fills
* s3api: align remote copy validators and reject stale ETags
* s3api: keep remote ETag through metadata touches, drop it after resizes
The remote-ETag check required Mtime to match RemoteMtime, so a utimens
on a cold remote object returned an empty ETag until a later cache fill
restored the remote value. It also trusted LastLocalSyncTsNs alone, so a
setattr resize that leaves chunks and the stamp untouched kept serving
the old remote ETag for different local bytes.
Require FileSize to match RemoteSize for both the synced and remote-only
paths and stop comparing modification times: content changes already
surface through cleared sync stamps, chunks, or inline content, while a
metadata-only touch leaves the remote bytes authoritative. Regression
tests cover touched cold and cached entries and resized cached entries.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix(s3api): accept max-keys up to 2147483647 and cap list pages at 1000
* add tests for ListObjects max-keys range and 1000 cap
* refactor(s3api): return uint16 from parseMaxKeys to avoid narrowing casts
* s3api: register the azurekms KMS provider under its build tag
weed/kms/azure registers itself with weed/kms from an init(), but the
blank import that triggers it was commented out in auth_credentials.go
(TODO: Fix Azure SDK compatibility issues). Nothing else imports the
package, so building with -tags azurekms still ends with
"KMS provider 'azure' not registered" at startup for any config using
"type": "azure".
Move the import into weed/s3api/kms_azure.go behind //go:build azurekms so
default builds stay free of the Azure SDK, and drop the dead comment.
Closes#11686
* ci: compile and test azurekms-gated code
weed/kms/azure is excluded from every default build, so nothing in CI
compiled it; that is how the provider shipped unregistered. Build the
tree and run the kms tests with -tags azurekms on every Go change.
---------
Co-authored-by: Yi-111-a <>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-10 23:44:09 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* worker: reject balance moves onto servers without the volume's disk type
Proposals are bucketed by disk type at detection time, but queued or
out-of-band jobs can still target a server that lacks the volume's disk.
The copy then reads the source disk only to be rejected by the target's
VolumeCopy, and the failure is retried on every scheduling cycle.
Fetch the master's topology once per job and check each move's target
against the volume's disk type before executing, so an incompatible move
fails before any data is copied.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* worker: cover the balance disk-type guard
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* worker: match old-topology nodes in the balance disk-type guard
Topology entries from older masters omit Address and carry a plain
host:port Id, so checkMoveDiskType never matched grpc-suffixed move
nodes and silently skipped the disk-type check. Also compare each node
via pb.NewServerAddressFromDataNode, which reconstructs the same
host:port.grpcPort form from Id and GrpcPort.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: apply bucket default encryption when volume data encryption is enabled
When -s3.encryptVolumeData (s3a.cipher) is enabled, putToFiler skipped
checking and applying bucket default encryption due to a '!s3a.cipher'
guard. Volume-level data encryption and object-level Server-Side
Encryption (SSE-S3 / SSE-KMS) operate at different layers, and explicit
SSE headers already work alongside volume encryption.
Remove the '!s3a.cipher' guard so PutObject without explicit SSE headers
inherits bucket default encryption regardless of volume data encryption.
Add regression test TestPutObjectAppliesBucketDefaultEncryptionWithVolumeCipher.
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* s3api: decrypt the volume cipher on direct SSE chunk reads
fetchFullChunk, fetchChunkViewData, and createEncryptedChunkReader fetched
raw bytes over HTTP, so volume-encrypted chunks reached SSE-S3/KMS/C
decryptors still ciphered. Route them through fetchChunkData: encrypted or
compressed chunks go through RetriedFetchChunkData (cipher-aware, slices
plaintext space for views); plain chunks keep the streaming range read.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: cover cipher-aware chunk reads with a fake volume server
The fake volume now serves stored GETs with Range support, and
TestFetchChunkDataDecryptsVolumeCipher verifies full-chunk and view reads
return plaintext for ciphered chunks while plain chunks still slice via HTTP
ranges.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: guard fake volume server stored map with its mutex
The HTTP handler goroutine read v.stored while test goroutines wrote
it, a data race go test -race can flag. Lock v.mu around the map read
and the test writes.
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* kms/azure: fix encrypt/decrypt round trip for Azure Key Vault
The provider stored the wrapped data key as string(encryptResult.Result)
in the JSON envelope, sent the encryption context as AAD to an RSA-OAEP
key, and passed a full key URL to the azkeys client in the name
position. Each of those breaks a round trip on its own.
- split the key URL into the (name, version) pair azkeys expects before
Encrypt, Decrypt and GetKey; a plain name still resolves to the latest
version, and decrypt keeps the stored version so objects stay readable
after a key rotation
- store the wrapped key base64-encoded and decode it strictly, so the JSON
envelope no longer replaces the raw bytes with U+FFFD
- stop sending AAD: Key Vault rejects it on RSA-OAEP with BadParameter,
so the encryption context is logged and dropped instead
* kms/azure: reject a key URL that names another vault
splitKeyID dropped the host of a full Key Vault URL, so a key ID from a
different vault silently resolved to this vault's same-named key. Return
an error instead of encrypting under a key the caller did not ask for.
* kms/azure: bind encryption context via an envelope digest
RSA-OAEP rejects AAD, so removing it left the encryption context
unauthenticated: a wrapped key could be decrypted under a different
object's context. Record a sha256 digest of the marshaled context in
the envelope's provider_specific field on encrypt and verify it before
calling Decrypt, restoring the binding without AAD.
* kms/azure: treat an explicit :443 port as the same vault
* kms/azure: accept an absent context digest only for empty contexts
* kms/azure: normalize both hosts when comparing key URLs to the vault
splitKeyID stripped :443 and a trailing dot from the configured vault but
only :443 from the key URL, so a key URL naming the same vault with a
trailing DNS dot was rejected before Azure was ever called. Compare both
sides through vaultHost so they are normalized identically.
* kms/azure: reject non-key vault URLs in splitKeyID, document digest limits
A Key Vault URL that does not name a key under /keys/, has an empty key
name, or carries extra path segments now fails fast instead of being
passed to the client as a key name, where it would surface as a
confusing vault-side error.
Also note that the envelope context digest is a client-side mismatch
check, not vault-authenticated AAD, and only allocate providerSpecific
when a context is present.
* ci: compile and test azurekms-gated code
weed/kms/azure is excluded from every default build, so nothing in CI
compiled it; that is how the provider shipped unregistered. Build the
tree and run the kms tests with -tags azurekms on every Go change.
---------
Co-authored-by: Yi-111-a <>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: fail the request, not the process, when chunks cannot be resolved
CompactFileChunks and ViewFromChunks discarded the error from
NonOverlappingVisibleIntervals and iterated the nil interval list it
returns on failure. A CreateEntry or UpdateEntry whose context was
cancelled therefore panicked in SeparateGarbageChunks and took the whole
filer down, losing the metadata log it had not yet flushed.
Both functions now return the error. cleanupChunks fails the request
with it, the mount flush paths return it, the replication sinks and the
S3 SSE range reader return it, and a ChunkStreamReader that cannot build
its view fails every read with it and reports it as its SourceError
instead of reading as an empty file.
Fixes#11682
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* filer: propagate chunk resolution failures in scan paths
NonOverlappingVisibleIntervals can fail without a cancelled context; the
query engine and log-store callers discarded the error and dereferenced
nil intervals.
* query: propagate chunk resolution errors instead of skipping files
When a manifest cannot be resolved, ReadParquetStatistics,
countLiveLogRowsExcludingParquetSources, and computeLiveLogMinMax logged a
warning and skipped the failed file, so queries returned understated counts
and wrong min/max values. Surface the errors instead: the fast-path
aggregation collector now returns a DataSourceError so the engine falls
back to a full scan, and the row-count helpers return the error rather
than a partial total.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(s3api): accept max-keys up to 2147483647 and cap list pages at 1000
* add tests for ListObjects max-keys range and 1000 cap
* refactor(s3api): return uint16 from parseMaxKeys to avoid narrowing casts
* fix(s3api): validate ListObjectVersions max-keys and honor max-keys=0
* add test for ListObjectVersions max-keys=0
* fix(s3api): widen uint16 max-keys for ListObjectVersions
2026-10-10 22:36:43 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: fail writes when the existing-entry lookup fails
CreateEntry only returned a FindEntry error for exclusive creates; a
regular write treated a transient store error as "entry not found" and,
on upsert stores, replaced the entry without loading its old chunks —
leaving them out of the cleanup process. Return any lookup error except
ErrNotFound for both paths.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: test that a lookup failure fails a regular write
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-10 22:32:23 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: count index-load metrics only for live-key removals
The in-memory and leveldb offset loaders counted a deletion for every
tombstone row replayed, including re-deletes of already-deleted keys
(runtime logDelete only counts a live removal), and added the returned
old size to the byte counters even when it was a tombstone sentinel —
uint64(-1) wraps DeletionByteCounter. The leveldb offset loader also
counted every index row as a file and its bytes unconditionally. Gate
the counters the same way the runtime paths do so a reload reports the
same numbers the runtime counters hold.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: derive index metrics from latest-row state in the metric loader
The leveldb metric loader walked the index in reverse and incremented
FileCounter for every row (inflating FileCount with tombstones) and
DeletionCounter for every superseded row plus every deleted-key row
(double counting both). Per key, the runtime deleted count is exactly
(valid rows) - (1 when the latest row is live), so count live-latest
rows via the same bloom filter and derive both deletion counters; this
matches the runtime logPut/logDelete counters exactly instead of
drifting with tombstone history.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* server: warn instead of failing volume copies on counter drift
checkCopyCounts compared the source's in-memory counters against the
target's fresh-load counters, but counters computed by different
loaders drifted apart for identical .idx data, so a good copy was
rejected (issue #11678). The file-size checks already gate integrity;
log the mismatch instead of failing the copy.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: assert MaxFileKey parity in the metric loader test
MaxFileKey derives from row order alone, so unlike the bloom-filtered
deletion counters it must always match the runtime counters exactly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A remote-only entry whose object was deleted from the remote storage
outside the filer answered GET with 500 and stayed in the filer. The
remote's not-found was lost on the way: the backends' ReadFile returned
it as an untyped error, so FetchAndWriteNeedle failed with codes.Unknown
and nothing downstream could tell it from any other failure.
- remote_storage: GCS, S3 (NoSuchKey) and Azure (BlobNotFound) reads
return ErrRemoteObjectNotFound. GCS reports a missing bucket the same
way as a missing object, so it confirms the bucket with a listing.
- volume server and filer: the not-found crosses gRPC as codes.NotFound
carrying the sentinel's text, and the filer's cache RPC returns
codes.NotFound, which the S3 gateway already maps to NoSuchKey.
- s3api: the origin fallback answers NoSuchKey on a confirmed not-found.
- filer: a confirmed not-found removes the stale entry, so the lazy
remote-metadata cache converges on the remote. Only remote-only files
outside .versions and without an active object lock are removed, only
if unchanged since the fetch (checked on the object's write owner,
under the lock S3 object writes take), and with a metadata-only
delete: the filer skips its inline remote delete, and the delete
events' entries carry a marker that makes filer.remote.sync and
filer.remote.gateway skip their remote delete, while filer.sync still
replicates it. The replicated DeleteEntryRequest carries
keep_remote_object, so the destination's delete events are marked too.
The store drops the marker from every write, so clients cannot plant
it.
2026-10-10 11:00:04 +08:00
Mohd Quamar TyagiGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: resolve volume data encryption in CopyObject SSE flows (#11646)
* s3api: honor bucket-default KMS key on copy and fix transformed-upload metadata
- Synthesize the destination bucket's default encryption as request
headers before any SSE evaluation, so the configured KMS key ID and
bucket-key setting reach the copy paths instead of only a boolean.
- uploadTransformedChunkData returns the upload result so callers record
the uploader's cipher key AND compression decision; a wrongly cleared
IsCompressed made transformed copies unreadable.
- decompressChunkVolumeCipher fails loudly when a compressed chunk does
not decompress, instead of uploading still-compressed bytes marked
uncompressed.
- copyMultipartSSECChunk now strips the volume cipher and re-encrypts on
upload like the other transform paths.
- The ciphered inner upload now uses the caller's private BytesBuffer.
* s3api: extract copy bucket-default header synthesis for coverage
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: test bucket-default encryption header synthesis on copy
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: validate copy encryption headers before bucket defaults and resolve empty KMS key
* s3api: name the AWS-managed SSE-KMS default key once
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-10 10:59:36 +08:00
tchobanskiGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Claude Opus 5.5Chris LuChris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(filer): end metadata subscriptions before gRPC GracefulStop on shutdown
GracefulStop waits for every open stream. The filer's own MetaAggregator
subscription and, under -s3, the S3 gateway's IAM subscription live in
the same process and only end when the filer shuts down, which happens
after GracefulStop - so every shutdown sat out the full 15s timeout.
StopSubscriptions ends them (and any started later) right after leaving
the lock ring; the handlers derive their context from the stream and
this signal. Measured on weed server -s3: stop 25.6s -> 10.7s, and
0.6s with -volume.preStopSeconds=0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FTq6bDpgdQfqQagQsUUvdw
* fix(filer): end stopped subscriptions with Unavailable, not cleanly
A subscription that StopSubscriptions cut short returned nil. The client
reads that as io.EOF, "caught up, done", and util.RetryUntil - which the
S3 gateway and mount follow with - stops on nil. A separate S3 gateway
therefore never resubscribed after its filer restarted (measured: 0
subscriptions in 25s after the restart; upstream master and this fix: it
is back within a second).
endOfSubscription now turns that clean end into codes.Unavailable, which
is what a dropped connection looks like, so followers reconnect. Errors
pass through, and so does the end of a stream the client closed.
A subscriber that has stopped reading is still left to GracefulStop's
timeout: its handler sits in stream.Send, which only the end of the
stream releases, and grpc-go does not allow Send after the handler
returns. Documented at StopSubscriptions. Shutdown under weed server -s3
is unchanged at 10.7s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FTq6bDpgdQfqQagQsUUvdw
* fix(filer): interrupt in-flight subscription reads and end them Unavailable
StopSubscriptions only reached the wait points: a handler replaying the
ring or the persisted log kept streaming until the pass ended, and a pass
cut short at a ctx.Err() boundary (the replay semaphore, chunk ref
collection, ref sends) returned a wrapped context.Canceled, which reads as
codes.Canceled on the wire - a code transient-error classifiers treat as
non-retryable.
eachLogEntryFn and sendRefsBatched now check the subscription context per
entry/batch, and endOfSubscription reports a context.Canceled cut short by
the stop signal as codes.Unavailable, same as a clean end it interrupted.
* filer: honor the subscription context in the pipelined sender
A subscriber that stops reading wedges sendLoop in stream.Send; Send and
Close then blocked past StopSubscriptions, so GracefulStop still waited out
its timeout. Both now return when the subscription context ends, letting the
handler return and gRPC tear the stream down.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: stop sendLoop from starting sends after the subscription ends
With Send and Close released by the subscription context, sendLoop could
still issue a stream.Send after the handler returned. Gate every send on
the context so at most the in-flight Send overlaps teardown - and the
stream's own teardown is what unblocks it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: assert sendLoop exits in the cancel test
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: consolidate session token extraction into extractSessionToken
Three call sites duplicated the same header/header/query lookup; share
one helper named after the existing s3tables equivalent.
* s3api: tolerate session tokens on statically configured credentials
Credential vendors like Unity Catalog emit a session token with every vended credential, including static ones (UC's StaticAwsCredentialGenerator only engages when s3.sessionToken is set). Requests signed by a configured access key were routed to STS validation and rejected, so static credential vending never worked against SeaweedFS.
Resolve the access key first: when it maps to a configured credential the signature alone authenticates the request, and the attached token is marked ignored so authorization does not route it into the STS session-policy path. STS-issued access keys are never in the static map, so temporary credentials still validate their token exactly as before.
* s3api: cover static credentials carrying a session token
* test: exercise UC static credential vending against SeaweedFS
s3.sessionToken.0 selects UC's StaticAwsCredentialGenerator, which vends the configured keys verbatim. The vended session token is foreign to SeaweedFS and previously failed SigV4; now it round-trips through temporary-table-credentials into real S3 I/O.
* s3api: reject temporary credentials in GetFederationToken after auth
Token presence alone cannot distinguish a temporary credential from a
statically configured one carrying a vended token. Move the check behind
verifyV4Signature and key it on the operative session token so tolerated
tokens keep the caller eligible.
* s3api: test GetFederationToken with authenticated temporary credentials
Sign the rejection cases with real session credentials so they reach the
post-auth check, and cover a vended static credential being accepted.
* test: check vended-credential delete error in UC integration test
* volume: answer chunk-manifest HEAD from the chunk-size sum
* volume: size chunk-manifest reads by the sum of chunk sizes
* volume: saturate the sum of chunk-manifest Range lengths
* volume: fetch remote chunk-manifest ranges with a bounded Range
* volume: fmt chunk-manifest handlers
* volume: cap whole-window chunk reads at the visible window
* volume: reject an over-cap chunk span and an illegal manifest MIME
* volume: take the manifest transform extension from the URL first
Go's tryHandleChunkedFile keeps the extension parsed from the URL —
including a fid suffix like /vid,fid.png — and only falls back to the
resolved filename's extension when the URL has none. The manifest path
now does the same, so a transform requested on a fid-suffixed URL runs
as it does in Go.
Also: compute the chunk-size sum once, drop an unused remote_read_for
parameter, share the crop/resize predicate with parse_read_request, and
settle a couple of clippy lints.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3api: revoke the identities a config reload no longer declares
A static config file reload merges with isFullState=false, so an identity
deleted from -s3.config kept authenticating with all of its access keys until
the process restarted. An operator revoking a leaked key got no error and no
revocation.
The file is the source of truth for the identities it declares, so a reload now
drops the ones that have left it, together with their access keys and those of
their service accounts. staticIdentityNames was only ever added to, which kept a
removed name protected as well; the names the file itself declares are tracked
separately from the AWS environment credentials, so a reload never revokes what
the file never declared.
* s3api: keep a file reload from replacing the dynamic store
Removing the last identity from a config file emptied staticIdentityNames, so
useStaticConfig turned false and the next reload of that file took the replace
path: every filer-managed identity and its access keys disappeared until a
dynamic reload brought them back.
A static config file is authoritative for the identities it declares and never
for the dynamic store, so a file load now always merges. The first file load at
startup takes the merge path as well, from an empty state.
Reported by the Devin and Greptile reviews of this PR.
TestReloadStaticConfigWithoutIdentitiesKeepsDynamic reloads an emptied file twice
and asserts that a filer-managed identity and its access key survive both; the
existing test now also checks that the service account key was loaded before the
reload and that the environment identity's key still works after it.
* s3api: clear the environment credentials in the emptied-file test
The AWS environment identity is static and is re-added after every merge, so on
a runner that has AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set it kept
hasStaticConfig true once the file was emptied, and the next reload still took
the merge path: the regression this test guards passed unnoticed.
Clear both variables whatever the runner has set. With the pre-fix condition
(hasStaticConfig alone) and the variables present in the environment, the test
now fails with "reload 2: a filer-managed identity must survive a reload of an
emptied file".
Reported by the Greptile review of this PR.
* s3api: verify under the write lock before re-committing an ambiguous routed PUT
A routed PUT whose response was lost after the owner committed, or whose
transaction returned a store error, fell straight into the lock path and
re-sent the same entry. A concurrent PUT that had superseded the commit in
the meantime had already deleted this entry's chunks as old, so the
re-commit restored metadata pointing at dead needles and the object read
back 404 permanently.
The lock path now resolves the routed attempt's outcome inside the write
lock first: a stored entry with the uploaded chunks means the route
landed; a different stored entry or an unresolved lookup refuses the
re-commit with ServiceUnavailable so the client retries with a fresh
upload; only a proven-absent entry falls through to the normal create.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: tighten ambiguous routed PUT recovery
- a match found only after an unanswered filer is not authoritative;
the unreachable filer may hold a newer entry, so refuse instead of
finalizing on it (both recovery paths)
- a failed post-recovery finalization now rolls the recovered entry
back, matching the create path's undo
- a proven-absent entry is only re-committed after probing that the
uploaded chunks' needles are still alive; a committed-and-deleted
PUT would otherwise write back an entry pointing at reclaimed
needles
- the .versions latest pointer no longer flips back to an older
version when a newer one committed while the write was uncertain
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: stamp late versions noncurrent when a newer latest pointer wins
The keep-newer early return in updateLatestVersionInDirectory left the
version just stored without ExtNoncurrentSinceNsKey, so the lifecycle
engine could never age it out.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: only a failure past mutation 0 makes a routed PUT ambiguous
A deterministic refusal at the PUT mutation (e.g. "existing entry is a
directory") applied nothing, but marking it ambiguous sent the lock-path
fallback through conservative recovery, which found the directory entry
and refused with 503 instead of the correct 409.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep all transaction errors ambiguous; route directory conflicts through the lock path
* s3: verify uploaded chunks before re-committing over a directory
A stored directory does not prove the routed PUT never committed: the
route can land, a delete can remove the entry and its chunks, and a
nested write can recreate the directory before recovery takes the lock.
Re-committing then stores an entry pointing at dead needles. Probe the
uploaded chunks first and refuse with ServiceUnavailable when they can
no longer be verified.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: skip cache lock when a cacher cannot be removed yet
SingleChunkCacher.readChunkAt defers removeConsumed on every read, and
every unpin retries removal too. Each call acquired the ReaderCache
mutex even though the removal conditions are plain atomics, so a busy
cache paid a lock acquisition per read just to discover readers>0.
Check the atomics before locking: when a cacher is not yet consumable
the removal is impossible and the lock round trip is pure contention.
Removals still run under the lock via removeConsumedLocked, so the
attach-vs-remove race keeps its existing serialization.
Ref #11676
* filer: guard stream position with a per-stream mutex, not the cache lock
chunkStream.cacher is only ever shared by concurrent ReadAt calls on one
ChunkReadAt, yet pin, unpin, releaseIfFinished, and releaseStream all
mutated it under the ReaderCache mutex that serializes every reader in
the process. Each read that switched chunks took that global lock to pin
the new chunk, released it, then reacquired it in unpin() just to drop
the old chunk's pin and retry removal; releaseIfFinished and
releaseStream did the same lock-detach-unlock-relock dance. At high
small-GET concurrency that turned per-request stream bookkeeping into a
global convoy (#11676).
Give chunkStream its own mutex and drop the cache lock from the pin
lifecycle entirely: pin/detach serialize on stream.mu, the pin counter
and consumable checks are already atomics, and removeConsumed only
takes the cache lock when a cacher is actually removable. The map lock
now guards only map membership and read registration.
* filer: look up cached chunks under the read lock
readChunkAt serialized every small read on the write lock even though
the common paths are read-only: an existing downloader just needs its
read registered, and a chunkCache hit needs no map access at all. Each
GET also paid the lock a second time to reach the chunkCache check.
Use the read lock for the downloader lookup and registration; the
registered read keeps the buffer alive against a concurrent destroy,
and only error eviction, insertion, and removal need the write lock.
The chunkCache probe runs lock-free, and the insert path re-checks the
map under the write lock to cover a downloader registered in between.
* filer: start the chunk download outside the map lock
The insert path held the ReaderCache write lock across goroutine spawn
and the cacheStartedCh handshake, so every downloader miss serialized
against the startup of a fetch goroutine. Register the cacher in the
map under the lock, then start the download after releasing it; a fetch
that fails early still lands in the map and is evicted by the next
reader's completed-error check.
* filer: test that stream pin lifecycle stays off the cache lock
Regression coverage for the contention fix: pin and releaseStream on a
non-removable cacher must complete while the ReaderCache lock is held
by another goroutine.
* filer: pin the stream under the map lock
Between read registration and stream.pin the cacher showed zero pins,
so a budget eviction in that gap could pick a chunk the stream was just
attaching to and the stream's next slice refetched it. The pin counter
is an atomic and stream.mu is never held while acquiring the cache
lock, so pinning inside the map hold is deadlock-free and closes the
window.
* s3api: reject SSE headers PutObject cannot honor, as CopyObject does
PutObject and CreateMultipartUpload did not check the server-side
encryption headers they were given. An x-amz-server-side-encryption
value that names no method, such as "aes:kms", matched none of the SSE
paths, so the object was stored unencrypted and the request answered
200. SSE-C together with x-amz-server-side-encryption was also accepted,
and one of the two silently won.
CopyObject already rejects both through validateEncryptionCompatibility.
Run the same check, with the same error codes, before PutObject and
CreateMultipartUpload store anything.
* s3api: answer rejected SSE headers with InvalidArgument, as S3 does
S3 rejects an unknown x-amz-server-side-encryption value and SSE-C
combined with another method with InvalidArgument. PutObject and
CreateMultipartUpload now return that code, with S3's messages, through
two new error codes; CopyObject keeps its own.
Also run ceph/s3-tests' test_put_obj_enc_conflict_c_s3, _c_kms and
_bad_enc_kms in CI, which check both handlers' responses end to end.
* s3api: close the remaining ways a PUT could skip requested encryption
- A repeated x-amz-server-side-encryption header is rejected: the
encryption paths apply only the first value, so extra values could
hide the method the client asked for.
- KMS options (key id, encryption context, bucket key) are rejected
unless the method is aws:kms, matching the S3 InvalidArgument error.
- CreateMultipartUpload validates before auto-create, so a refused
upload cannot leave a bucket behind.
- Directory markers encrypt their inline content through the shared
SSE path and record the same entry metadata as regular objects,
instead of storing requested-encrypted bytes in plaintext.
* s3api: read back what the SSE marker write stores
- Repeated SSE-C and KMS option headers are rejected alongside a
repeated x-amz-server-side-encryption, closing the same first-value
bypass for customer-key and KMS fields.
- Algorithm validity is checked before KMS options so an unsupported
value keeps the "not supported" error.
- serveDirectoryContent decrypts marker content with the stored SSE
metadata and returns the SSE headers, so an encrypted marker reads
back what was written; its Content-Length now reflects the bytes
actually served.
* s3api: HEAD of a marker skips decryption and keeps the stored size
HEAD returns no body, so it now runs only the SSE-C key check instead of
decrypting — matching HeadObjectHandler and avoiding a KMS round-trip —
and chunk-backed directory entries report Attributes.FileSize again
rather than the length of their (empty) inline content.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: keep GET Content-Length to the bytes serveDirectoryContent writes
The FileSize override described chunk-backed markers on HEAD, but GET
sends only the inline content, so it promised bytes it never wrote.
* s3api: stream a chunk-backed directory on GET like any object
A directory promoted over an uploaded object keeps the object chunks
with empty inline content, so serving only entry.Content made GET
deliver nothing while HEAD reported FileSize. GET now routes those
entries through the regular volume-server stream, keeping the two
methods consistent.
* s3api: answer a busy volume read with RequestBytesExceed
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* helm: add Gateway API HTTPRoute for master, volume, filer, s3 and admin
Each component that offers an Ingress can now be exposed through a
Gateway API HTTPRoute instead, via <component>.httpRoute. It is disabled
by default, so existing renders are unchanged.
parentRefs, hostnames, labels and annotations pass through as given.
Each rule may set matches, filters, timeouts and backendRefs, and a rule
without backendRefs routes to the component's own service and port,
following all-in-one mode the same way the Ingress templates do. An
empty rules list yields a single rule sending all traffic there.
Signed-off-by: younsl <cysl@kakao.com>
* helm: fix HTTPRoute backend selection for S3 on filer and all-in-one
- s3: with S3 on the filer the route targets filer.s3.port, the port the
s3 Service exposes. The all-in-one Service is chosen only when
allInOne.s3 is enabled, so a standalone S3 next to all-in-one routes
to the s3 Service.
- master: render in all-in-one mode and route to the all-in-one Service,
like the volume and filer routes.
- Name routes with seaweedfs.componentName, keeping them within 63
characters like the Services they point to.
- CI: cover the filer S3 port and the all-in-one master route.
Signed-off-by: younsl <cysl@kakao.com>
* helm: document the Gateway API HTTPRoute values in the chart README
Signed-off-by: younsl <cysl@kakao.com>
* helm: link the Gateway API docs from the chart README
Signed-off-by: younsl <cysl@kakao.com>
* helm: scope the filer S3 port note in the chart README
Signed-off-by: younsl <cysl@kakao.com>
---------
Signed-off-by: younsl <cysl@kakao.com>
2026-10-09 11:37:10 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The handler sends the header non-blocking and the test stops on fetch
errors instead of waiting on a channel that may never be fed.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-09 10:41:49 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: stop the compat server even when startup fails
test-with-server depended on start-server as a prerequisite, so a failed
startup skipped the recipe and the spawned weed mini kept the port.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: verify the recorded PID is weed before stopping it
A prerequisite failure leaves a stale weed-server.pid in place; the PID
may have been reused by an unrelated process.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-09 10:01:53 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
calculateSeedSignature called verifyV4Signature with
shouldCheckPermissions=true, which runs VerifyActionPermission — a
check that does not consult bucket policies. Every caller of
newChunkedReader is already behind the Auth middleware, which does
evaluate them, so a principal allowed only by the bucket policy was
authorized upstream and then denied when the handler built the body
reader: aws-chunked PutObject/UploadPart (botocore's default shape
over TLS, PyArrow's over HTTP as well) failed with AccessDenied.
The reader now verifies only the signature; a wrong secret is still
SignatureDoesNotMatch. The non-streaming paths already work this way.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-09 10:01:28 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
rewrite_position_delete_files emitted a schema-less parquet file whose
file_path column was DELTA_LENGTH_BYTE_ARRAY. PyIceberg reads delete
files with file_path as a dictionary column, which PyArrow cannot
decode from that encoding, so every table the worker rewrote failed
scans outright; readers that resolve columns by field id found none.
The row struct now declares the spec's reserved field ids
(2147483546 file_path, 2147483545 pos) and dictionary-encodes
file_path, which is also the compact choice for a column repeating one
data file's path.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-09 10:00:48 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Orphan cleanup built its referenced set from snapshot manifests plus the
current and previous metadata files, then walked all of metadata/ and
data/. Table statistics (Puffin) and partition statistics files are
listed in the table metadata's statistics and partition-statistics
lists, not in any snapshot, so once they passed the safety window
remove_orphans deleted them while the metadata still pointed at them.
Mark the files in both lists referenced as well.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-09 09:59:58 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: answer a denied write with 403 when the caller can read the entry
Update/Delete/Rename answered every authorization refusal as not-found so
the denial leaked no existence signal. For a caller allowed to GetTable
(or GetView) the same entry, the veil hides nothing it could not load —
yet a refused write was still answered 404, so clients saw a table they
just loaded reported as missing.
When the write check fails, re-check read permission on the same entry:
readable entries get 403 AccessDenied; invisible ones keep the
not-found answer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg: map UpdateTable errors through writeManagerError on commit paths
CommitTable and CommitTransaction answered any non-conflict UpdateTable
failure, including AccessDenied and NoSuchTable, with a bare 500. A 500
reads as outcome-unknown to clients (PyIceberg raises
CommitStateUnknownException) where a refused commit is a plain
ForbiddenException, matching what the drop and rename handlers already
emit through the same mapper.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: keep the not-found veil over views and match the real read check on renames
UpdateTable and DeleteTable read shared metadata without checking the
entry kind, so a denied write on a view reported 403 to a GetTable-only
caller where a missing name reports 404. The rename visibility check
also fed resource tags into GetView evaluation that the real GetView
path never supplies.
* s3tables: gofmt handler_table.go
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(volume): keep the TTL clock across a vacuum commit instead of rescanning
CommitCompact reloads the swapped files while holding dataFileAccessLock,
and for a vacuumed TTL volume that reload re-derived lastModifiedTsSeconds
by reading every live needle's append timestamp from the .dat: two random
reads per needle, with every read of the volume blocked behind them.
The in-memory clock is already current at that point. Every write since the
volume loaded moved it, and makeupDiff only replays writes that went through
that path. Carry it across the reload instead. This also stops an
over-budget scan from falling back to the new .dat's mtime and restarting
an expiring volume's TTL at the commit.
* volume: carry the append watermark as the TTL clock across a vacuum commit
The running append watermark is the clock the reload's recovery scan
recomputes, so the commit can keep it directly. Client-supplied needle
modified times can run ahead of or behind the append time; keeping
lastModifiedTsSeconds itself would let a forged or stale timestamp move
expiry through a vacuum, where the scan it replaces used server-side
append timestamps.
* storage: test that a vacuum commit keeps the append clock
A write's client supplied modified time can lie ahead of or behind its
append time; the commit must land the TTL clock on the append watermark,
the same value the recovery scan would have recomputed.
* volume: carry the append watermark as the TTL clock across a vacuum commit
Mirrors the Go volume server: the running append watermark is the clock
the reload's recovery scan recomputes, so the commit keeps it instead of
rescanning live needles under the write lock.
* volume: commit carries the last-write append time, not the latest append
lastAppendAtNs counts tombstone appends and is reseeded from the .dat
tail at every load, so it can sit ahead of the last write -- a delete
freshens the commit clock -- or behind it: a restarted vacuumed volume's
tail needle is not its newest write, and the commit would move the TTL
clock backward into premature expiry.
Track lastWriteAppendAtNs instead, bumped only on needle appends and
seeded by the recovery scan, so the commit lands the clock on the same
live-write maximum the rescan would have recomputed.
* volume: rescan at commit when the newest write was deleted
lastWriteAppendAtNs can hold a write the index no longer holds, so
carrying it extends the TTL clock past what recovery over the compacted
index would compute. Remember the key behind the watermark so its
tombstone or index rollback can send the reload back through
recoverLastModifiedTs, landing on the newest surviving write.
* volume: a tombstone retires the write rows beneath it in the last-write scan
A needle deleted after the compaction copy leaves its write row followed
by a tombstone in the committed index. The reverse scan skipped the
tombstone row but then counted the dead write, reseeding the watermark
and flag as if it were alive. Track keys whose latest row is a tombstone
so their earlier write rows stop counting, and exercise the
delete-inside-the-commit-window ordering in the tests.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-08 22:58:44 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: record column statistics on compacted files
A compacted file's manifest entry was built with no column_sizes,
value_counts, null_value_counts, lower_bounds, upper_bounds or
split_offsets, so no reader could skip a compacted file on any
predicate. parquet-go already writes exact per-chunk min/max and null
counts into the footer; read that footer back after the merge and
record it on the data file, bounds as the spec's single-value
serialization with string and binary truncated as truncate(16). A
column whose bounds cannot be converted exactly gets none, and a
statistics failure is logged while the compaction commits anyway:
metrics are an optimization, not a correctness requirement.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: merge bins in bound order so compacted files stay prunable
A bin's files were concatenated in manifest order, or largest-first
when a partition was split under the target size, so inputs disjoint
on a column came out as outputs that overlapped on it. Order each
bin's files by their bounds on one column before merging and split an
oversized partition into runs of consecutive files, so every output
covers one contiguous range. The column is the first identity field
of the table's sort order when it declares one (a descending order
sorts by upper bound), otherwise the first schema column every
candidate file has bounds for; detection resolves the same order so
it plans the bins execution builds. Files without bounds keep the old
behavior.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: cap compacted files' row groups from table config
Neither merge writer set a row-group limit (parquet-go's default is
unlimited rows), so every compacted file was a single row group and
readers could not skip inside it either. Rows per row group now come
from the table's write.parquet.row-group-limit and
write.parquet.row-group-size-bytes, defaulting to PyIceberg's
1 048 576 rows and Iceberg's 128 MiB, with the byte size turned into
rows from the bin's inputs' compressed bytes per row and a floor of
1 024 rows. Both writers take the cap, and the statistics the entry
records list one split offset per row group.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg: keep compaction order eligibility per group, rescue stranded runs
The merge order resolved over all candidates, so one oversized or
non-Parquet file without bounds disabled ordering for files that could
participate. Resolve it per partition group over the eligible entries.
Ordered runs too short to merge were dropped entirely. Runs from an
inferred bounds order now fall back to size-based packing — ordering is
a preference there — while runs under a declared sort order are still
left for later passes so the sort contract holds.
An explicit write.parquet.row-group-limit is a cap, not a floor: values
below the estimate floor are now honored instead of being raised.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg: only use the declared sort order when its bounds are complete
An entry without bounds on the sort column sorted to the tail and merged
into an output claiming an order it cannot verify. Fall back to
bound inference instead of ordering by a later sort field alone.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: keep ordered runs when the full repack yields nothing
The bestEffort leftover fallback removed the ordered runs before checking
whether repacking the whole bin produced any bins, discarding valid
compaction work.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-08 22:06:44 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec: strict_placement option so encode only runs while guarantees hold
Shard placement during encode was best-effort (PlaceDurabilityFirst):
when the cluster could not satisfy the per-disk caps, anti-affinity,
replica-placement or per-rack caps, the constraints were relaxed and
the volume was encoded anyway, weaker than configured. A
strict_placement option on the erasure coding task switches planning
to PlaceStrict so the volume's planning fails instead, and the encode
is retried when capacity allows the guarantee.
Also documents the resilience rule in ec.encode help: a volume
survives losing any nodes or racks holding at most parity-shards
shards between them, and how -shardReplicaPlacement's rack and node
digits bound that loss.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec: expose strict_placement through the plugin form and persisted task policy
The admin UI, the admin.toml maintenance mapping, and the TaskPolicy
serialization all dropped the new flag; add the bool field to
ErasureCodingTaskConfig, the worker config form, and both conversion
directions.
* shell: describe shardReplicaPlacement as requested limits, not guarantees
ec.encode places shards best-effort, so the configured rack/node caps only
bound shard loss when the final placement actually satisfies them.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-08 22:05:02 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* deps: pin github.com/seaweedfs/goexif back to v1.0.3
The fork's newest tagged release is v1.0.3; the v2.0.0+incompatible
requirement resolved to older, untagged code and breaks isolated
builds that fetch it from the proxy.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: stop dependabot bumping seaweedfs/goexif past its latest tag
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/kafka: settle goexif at v1.0.3 too
The kafka test module recorded v2.0.0+incompatible in its own requires,
so MVS kept selecting it over the pinned v1.0.3 in the root module.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-08 22:04:36 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.sync: sign proxied chunk I/O from the per-side security file
The -a.security / -b.security files were used for gRPC TLS and the
HTTPS client but not for jwt.filer_signing, so filer-proxied chunk
reads and writes carried a token signed with the process-wide key and
failed authorization whenever the two clusters' keys differ.
LoadFilerJwtFromFile returns a FilerJwtProvider for each side's file,
which FilerSource and FilerSink now accept for proxied chunk reads and
writes. With no keys in the file or no flag, both fall back to the
process-wide jwt.filer_signing configuration as before.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: use the side filer read key for manifest downloads and fall back per access level
Manifest chunk resolution still signed proxied downloads with the
process-wide read key, so a source filer requiring its own key 401'd on
manifest-bearing files. A side security file that set only one access
level also produced empty tokens for the other instead of inheriting the
process-wide key, and the side file loader ignored the WEED_ environment
overrides the filer itself honors.
ResolveChunkManifest/ResolveOneChunkManifest keep their signatures;
FilerJwt-aware variants thread the provider down to fetchWholeChunk,
which prefers it on proxy URLs. The side loader now applies the same
environment precedence and falls back to the process-wide signer per
missing access level.
* security: verify the configured filer token lifetimes
* security: reject negative filer token lifetimes
A negative expires_after_seconds reached GenJwtForFilerServer and produced
a token with no expiration claim. Also synchronize the Authorization-header
capture in the proxy test and restore the prior viper key on cleanup.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-08 22:04:06 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(filer): gate the aggregated metadata disk pass on real change
A subscriber whose start position is past the end of the local persisted
log re-ran the whole persisted-log pass - store listings, file opens,
readahead - on every loop iteration. Each iteration is paced only by the
shortest wake (the 20ms hold floor on a busy watermark), so one parked
subscriber kept a full CPU core busy for the life of the stream.
The aggregated loop now mirrors the local loop's gate: the disk pass
runs on the first pass and afterwards only when something it cannot
miss changed - a local flush landed, the peers' flush low-watermark
advanced (more content admitted, or new files in a shared store), the
cursor moved, or a disk hold is pending (the ring read that follows an
empty pass parks internally, so skipping there would strand a held
entry).
Regression test: a subscriber parked past the persisted-log tail holds
the listing rate near zero and still delivers once peers report
progress.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: re-arm the aggregated disk pass on unobserved change
Review found three staleness classes the gate could not see: the flush
low-watermark only catching rises (a joining peer lowers the minimum and
invalidates an earlier pass's proof), a peer past the minimum landing a
file without moving it, and a chunk subscriber's refs-stop bound
advancing with wall time. Re-read when the low-watermark moves in either
direction, when the chunk listing bound admits more files, and on a slow
re-probe cadence for files no watermark can signal.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: unwind the parked ring read so the disk re-probe runs, and re-read on cursor rewinds
A caught-up subscriber parks inside LoopProcessLogData's wait loop, so
the re-probe interval in the outer disk gate could never elapse there;
the callback now unwinds the read once the cadence is due so the gate
re-evaluates. The cursor trigger also needs to notice rewinds, not just
advances, since ResumeFromDiskError moves the cursor backward.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3err: add InvalidArgument and AuthorizationHeaderMalformed codes
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: answer unrecognized bucket PUT sub-resources with 501
A PUT on a bucket carrying an unrecognized query (logging, metrics,
intelligent-tiering, ...) fell through to the bare CreateBucket route
and returned BucketAlreadyOwnedByYou or re-created the bucket. AWS
answers these with NotImplemented.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: reject malformed copy-source and multipart PUT parameters
A malformed X-Amz-Copy-Source or a non-numeric partNumber fell through
to the plain PutObject route and stored the body as a regular object.
Answer them with InvalidArgument-class errors instead of writing data.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: verify x-amz-content-sha256 against the streamed body
A PUT carrying a hex or base64 payload hash now streams through a
verifier that reports a mismatch once the stream is exhausted, instead
of storing an object that does not match its declared hash. The error
is deferred so intermediate reads that drop (n>0, err) results cannot
silently swallow it. Sentinel values (unsigned/streaming payloads)
remain exempt and malformed values fail fast.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: map truncated SigV4 headers to the error for the missing field
AWS answers an Authorization header missing Credential= with
InvalidArgument and one missing or malformed Signature= with
AuthorizationHeaderMalformed, instead of a generic MissingFields.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: answer IAM/STS failures in the query-protocol envelope
Embedded IAM and STS routes now report authentication, form-parse and
authorization failures with the IAM ErrorResponse body instead of the
S3 Error envelope, so IAM SDK clients can parse them. Requests signed
for s3 keep the S3 envelope, keyed off the credential scope.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: return 403 AccessDenied when the request has no Date
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: answer throttling rejections as SlowDown
ErrTooManyRequest and ErrRequestBytesExceed reported made-up codes;
AWS serves these throttling rejections as SlowDown.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: reject versionId requests on buckets that never had versioning
GET, HEAD and DELETE carrying a non-empty versionId on an unversioned
bucket now fail with InvalidArgument instead of being answered as a
plain object request.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: answer DeleteObjects over 1000 keys with MalformedXML
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: InvalidArgument for non-numeric or out-of-range part numbers
partNumber=abc, 0 and >10000 all resolve to InvalidArgument, matching
AWS, instead of InvalidPart.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: correct LimitExceeded, InvalidAction and ServiceFailure mappings
LimitExceeded is a conflict (409), an unknown Action is InvalidAction
(404) rather than NotImplemented, and internal failures report the IAM
receiver fault type. Applies to both the embedded IAM endpoint and the
standalone iamapi server.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: refuse DeleteUser while access keys remain
Deleting a user with live credentials orphaned its access keys; AWS
answers DeleteConflict until they are removed first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/s3: add S3/IAM error-response compatibility harness
* s3: return InvalidArgument for malformed x-amz-content-sha256
A header value that decodes to neither 32-byte hex nor base64 is a
malformed argument, not a hash mismatch.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: stop spawned mini when readiness times out
A slow-starting server otherwise survives the failure path and keeps
the S3 port occupied for the next run.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: register atexit cleanup before setup
A failed setup previously skipped cleanup, leaking the bucket and IAM
user on persistent servers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: reject unrouted subresources on the DELETE bucket catch-all
PutBucketHandler gained the same guard when the route-level check moved
into the handlers; DeleteBucketHandler was missed, so an authorized
DELETE /bucket?logging could delete the bucket instead of answering
NotImplemented.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: tolerate unset fixture variables in cleanup and cover DELETE ?logging
Cleanup now runs its IAM/multipart steps only when setup reached them, so
an early setup failure still removes the bucket. Added a DELETE
bucket-subresource case asserting NotImplemented and bucket survival.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: fail a case when its side-effect check reports a regression
A non-empty check note now fails the case, so a deleted bucket or an
object created by a malformed request cannot slip through behind a
passing status check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: give each prefix its own hidden-prefix probe budget
The shared cursor.probedEntries counter in dirHoldsOnlyHiddenEntries was
exhausted by one large all-deleted subtree, causing every later prefix in
the same request to be treated as visible. The fix allocates a fresh
budget (hiddenProbePerPrefixBudget, default 1000) for each top-level call
and threads it down to recursive calls via a pointer, so cross-prefix
budget bleed is impossible.
Fixes#10847.
* s3: keep 10000 probe budget, now per prefix
Restore hiddenProbeBudget = 10000 as a package const (not a mutable var)
applied per-prefix instead of per-request. The override hook moves to an
unexported probeBudget field on ListingCursor; zero means use the package
default. Tests set probeBudget: 3 on the cursor, keeping them fast without
touching package state. Also revert the unrelated uint32 cast on the
ListEntries Limit field.
* s3: cap total hidden-prefix probe work per listing request
The per-prefix budget resets for every candidate prefix, but deleted
prefixes do not spend maxKeys, so a page can walk an unbounded number of
them. Keep a request-wide probedEntries ceiling (hiddenProbeTotalBudget,
10x the per-prefix budget) so total probe work stays bounded.
* s3api: pin the probe-budget cutoff and the request-wide ceiling
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* ecbalancer: cap total shards per rack in Plan
Plan caps data and parity per rack separately (ceil(data/racks) and
ceil(parity/racks)), so with 10+4 over 8 racks a rack can legally hold
2 data + 1 parity. When a rack is one disk, losing two such racks loses
6 of 14 shards and the volume can't be read.
The cross-rack phase now also caps each rack's TOTAL shards of a
volume, sized with Place's rackTotalCap: ceil(shards/racks) unless the
racks lack room, counting a rack's own shards of the volume as room
since Plan can move them.
- A rack above the cap sheds parity until it fits. Those shards may go
to a data-bearing rack, and when no rack is under the parity cap
they fall back to a rack under the total cap. #11438's non-overflow
candidates keep moving only to data-free racks.
- No cross-rack move lands on a rack at the cap.
- A rack above the cap triggers balancing regardless of the imbalance
threshold.
- The fallback applies only while the source rack is above the cap;
otherwise the next Plan moves the shard back.
- Options.RackTotalCapRaised reports volumes whose cap had to be raised
above the even share; the worker and the shell log it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* ecbalancer: finish the rack cap in one Plan, count SameRackCount room
Review follow-ups.
- The data pass can run out of destinations before the parity pass frees
slots elsewhere, which left a data-heavy rack above the cap after one
Plan (a one-shot shell balance stops there). The cross-rack phase now
repeats while a rack is above the cap and the last round moved
something. A shard moves at most once per plan, since each move runs
as its own task.
- planRackTotalCap bounds each node's room by what SameRackCount still
allows, as Place does. Counting the raw free slots sized the cap too
low, so RackTotalCapRaised missed volumes it should report.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(s3): make self-heal pointer persist CAS-bound against concurrent writers
Follow-up to #11618: pointerless reads of a slash key whose regular-path
entry is a physical parent (or a bare-key object) now fall through to
healStaleLatestVersionPointer, which rescans .versions and persists a
repaired pointer. The persist was an unconditional upsert off the pre-scan
snapshot, so a PUT or delete that atomically advanced the pointer on the
owner filer while the heal was rescanning could be rolled back, making
older content or ACLs current again.
Mirror the CAS discipline clearStaleLatestVersionPointer already applies:
re-fetch the live .versions entry, require its pointer fields to still
match the ones the heal observed, and abandon the persist (still returning
the rescanned entry) when a concurrent writer has moved them. Write the
live Extended map so concurrently updated fields are preserved.
* fix(s3): close the check-then-act window in the self-heal pointer persist
The CAS re-fetch added in the previous commit narrows the race but leaves
a gateway-side window: after the live .versions entry is re-read and the
pointer compared, the repair is still written back through an
unconditional RPC, so a PUT or delete committing between the re-fetch and
the persist still ends up rolled back by the stale repair.
Bind the persist to the live image the heal just re-read with an
IF_ENTRY_EQUAL precondition, the same discipline routedSelfCopy applies
to stale self-copies: the filer evaluates the condition under the entry's
path lock and conditional writes route to the owner filer, so a writer
committing inside the window fails the precondition and the winner's
pointer stands. FailedPrecondition and NotFound are authoritative replies
and are not replayed by the failover layer.
The test now also covers a writer committing during the persist, which
reverts the pointer on the previous unconditional write-back.
* s3api: CAS-bind the stale-pointer clear against concurrent writers
The pointer clear re-read the live .versions entry and then wrote it
back unconditionally through mkFile, so a writer committing between the
re-fetch and the persist was rolled back to a cleared pointer. Persist
through the same IF_ENTRY_EQUAL conditional update as the repair path.
* s3api: test the CAS contract on the stale-pointer clear
* s3api: never clear a pointer the clear did not observe as stale
The CAS clear skipped its live-pointer match when the caller's snapshot
carried an empty latest-version id, so a writer promoting a version
between the clear's rescan and its re-fetch had the fresh pointer
CAS-cleared away (expected = the writer's own live entry), briefly
making the just-written version appear absent. With an empty observed
id, reaching the persist at all implies a concurrent promotion (an idle
key short-circuits as already-clear), so make the pointer match
unconditional and abort instead.
Extend TestClearStaleLatestVersionPointerConcurrentWriter with
pointerless-snapshot cases: a post-rescan promotion must survive, and an
idle pointerless key must short-circuit as already-clear. The fake
filer's proto round-trip drops empty Extended maps, so the snapshot is
padded the way real callers do.
---------
Co-authored-by: zhaoyuchen <yc.zhao@yinzon.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix(cosi): add get/list/watch on bucketclasses to enable static provisioning
* fix(cosi): bind provisioner deployment to the chart-created service account
The deployment referenced a release-prefixed service account name while the
chart creates and binds "seaweedfs-objectstorage-provisioner", so the RBAC
grants never reached the provisioner pod.
---------
Co-authored-by: Jonas Onshuus <jons@dips.no>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3api: extract quoteETag from setEtag
Consolidate ETag quoting so the XML response builders can share it.
* s3api: quote ETag in CopyObjectResult XML
AWS returns the ETag quoted in the copy result body, matching the ETag header. Fixesseaweedfs/seaweedfs#11622
* s3api: quote ETag in CopyPartResult XML
UploadPartCopy returned the raw ETag in the XML body while the response header and other APIs return it quoted. Fixesseaweedfs/seaweedfs#11622
* s3api: test ETag quoting in copy responses
* s3api: assert quoted ETag on the wire in copy response tests
* s3api: use strconv.Quote in quoteETag
Addresses CodeQL 'potentially unsafe quoting' on string concatenation.
* volume: return error instead of panicking when .dat open fails
When backend.OpenVolumeFile returns an error (e.g. disk below
-minFreeSpace), dataFile is nil. Calling backend.NewDiskFile(nil)
immediately after caused a nil-pointer panic in f.Stat()/f.Name().
Move the existing error check to run right after OpenVolumeFile, before
NewDiskFile is called, so the error is returned cleanly.
Fixesseaweedfs/seaweedfs#11615
* volume: share .dat load error handling
Extract datFileLoadError helper so open and create paths share one check.
* volume: trim dat-open-fail test comments
* volume(rust): cover unopenable .dat load path
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* build(deps): bump github.com/apache/iceberg-go
Bumps [github.com/apache/iceberg-go](https://github.com/apache/iceberg-go) from 0.6.1-0.20260817192109-c2105090c9e2 to 0.7.0.
- [Release notes](https://github.com/apache/iceberg-go/releases)
- [Commits](https://github.com/apache/iceberg-go/commits/v0.7.0)
---
updated-dependencies:
- dependency-name: github.com/apache/iceberg-go
dependency-version: 0.7.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* iceberg: write delete manifests with NewManifestWriter
iceberg-go 0.7.0 validates that delete entries only land in
delete-content manifests, so the WriteManifest + byte-level content
patch workaround no longer works: WriteManifest always creates a
data-content writer and rejects delete entries outright.
Use NewManifestWriter with WithManifestWriterContent instead and
dispatch entries through Add/Existing/Delete by status. The returned
ManifestFile now carries writer-computed counts, partitions and
min-sequence-number, so the manual ManifestFile rebuild is dropped.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: drop fromToml dependency in security-configmap.yaml (fixes#11611)
fromToml was added to Helm in v3.17.0 (helm/helm#12026, merged
2024-09-12, one day after v3.16.0 was cut). This chart declares no
minimum Helm version (no Chart.yaml kubeVersion, nothing in the
README), and the call in security-configmap.yaml:21 is an
unconditional *parse*-time failure on Helm < v3.17.0 - Go's
text/template parses a file's entire body before evaluating any
{{if}}, so this breaks the chart (any topology, any values) even when
securityConfigEnabled is false and the ConfigMap would render nothing.
Replaces the fromToml-based dig lookup with a small regex-based helper
(seaweedfs.existingTomlKey) that reads the same four "key = ..." JWT
signing-key values out of a previously-rendered security.toml,
preserving the existing fallback-to-random behavior exactly.
Verified:
- helm lint (v3.16.3 and v4.3.0): clean
- helm template with chart defaults: byte-identical output to the
unpatched chart rendered via Helm v4 (which has fromToml) - the
disabled/default path is untouched
- helm template with security enabled, no prior ConfigMap: identical
structure to the unpatched chart (helm v4), modulo the expected
random key
- Real helm install + helm upgrade round trip (live lookup, since
"helm template" never evaluates lookup, even under the original
fromToml code): the JWT signing key is identical across both
releases - confirms key persistence across upgrades is preserved,
not just "renders without erroring"
- helm template with chart defaults, Helm v3.16.3: previously failed
with a parse error naming fromToml as undefined; now renders
successfully
Fixes#11611.
* helm: harden existingTomlKey against commented key lines and CRLF
Addresses two review findings from greptile-apps on PR #11614:
- The key-line regex matched the first "key = ..." anywhere in the
section block, including a commented-out "# key = ..." line, which
would shadow a real active key on a hand-edited or otherwise
non-chart-generated ConfigMap. Anchored to line start with the
Go regexp multiline flag ((?m)^key...), which a line starting with
"#" cannot match.
- The section-header match required an exact "]\n", so a ConfigMap
with CRLF line endings would fail to match the block at all and
regenerate the key instead of reusing it. Changed to "]\r?\n".
Also adds a CI test ("Verify JWT signing key persistence across
upgrades") exercising all of this end to end with a real
helm install -> edit the live ConfigMap -> helm upgrade cycle,
matching the existing "Verify SFTP host key secret lifecycle" test's
shape: both edge cases are reproduced against a real ConfigMap and
asserted on the post-upgrade rendered security.toml.
Verified locally (same commands as the new CI step) against a real
cluster before pushing.
* helm: preserve JWT keys across supported TOML layouts
* ci: use setup-python interpreter for JWT upgrade checks
* helm: preserve keys under quoted TOML section headers
* helm: ignore unrelated quoted TOML section headers
The standalone s3 Deployment hardcoded terminationGracePeriodSeconds and
offered no way to set a container lifecycle or the Deployment strategy, so
operators could not add a preStop delay to drain endpoints before SIGTERM
or hold maxUnavailable at 0 during rollouts.
Add s3.updateStrategy, s3.lifecycle and s3.terminationGracePeriodSeconds.
Defaults render the same manifest as before.
Standalone weed s3 only shut its servers down when shutdownCtx was set,
which only weed mini does. On SIGTERM the interrupt hooks ran and the
process exited with requests still in flight, so clients saw connection
resets during rolling restarts.
Register an interrupt hook that drains every S3 HTTP(S) listener
(including the local and unix socket ones, and the Iceberg and Lance
servers) for up to 15s alongside a bounded gRPC GracefulStop, then closes
the S3 API server, reusing the filer's shutdown helper. Serve exits now
join that shutdown so the process does not exit mid-drain, and the
secondary listeners tolerate http.ErrServerClosed instead of exiting
fatally.
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-06 10:10:37 +08:00
Eliah RusinGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Claude Opus 5.5Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: read GET/HEAD needles off the store lock, and only once
The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.
The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stream needle chunks without the store lock
StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.
The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.
The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: split get_or_head_handler_inner into phases
get_or_head_handler_inner was a ~650-line function. Its middle resolved
the needle and set five mutable flags (stream_info, can_stream,
can_handle_head_from_meta, can_handle_range_from_source, bypass_cm) that
three if-let reply paths then re-tested, each re-checking stream_info.
It is now a 126-line orchestrator over named phases: reject_read_jwt,
proxy_missing_volume, wait_for_download_slot, parse_read_request,
read_ec_needle / read_volume_needle, etag_and_last_modified,
not_modified_response, read_response_headers, and the reply phases
stream_response, head_from_meta_response, range_from_source_response,
buffered_payload and buffered_response. The read phases return a
ReadPlan whose ReadStrategy enum (Stream, HeadFromMeta, RangeFromSource,
Buffered) carries the NeedleStreamInfo only on the variants that use it,
so the reply is one match instead of three flag checks.
Pure refactor: every status code, header and header order, error text,
metric increment, lock and data-file lease scope, spawn_blocking
boundary and side-effect order is unchanged. Phases that can end the
request return ControlFlow<Response, T>. A Range header that is not
visible ASCII still falls through to the buffered path, as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stop a needle stream once its volume becomes unavailable
Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.
The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: read a non-ASCII or empty Range header as Go does
A Range value with a byte >= 0x80 (obs-text, which hyper accepts) failed
HeaderValue::to_str at both range gates. For a needle served as stored the
handler had already chosen a meta-only read, so it fell through to the
buffered path with no payload and answered 200 with an empty body; a
compressed or EC needle answered 200 with the full body. Go's parseRange
fails on a byte it can neither trim nor parse and answers 416 "invalid
range", and trims Unicode whitespace such as NBSP into a normal 206. An
empty Range value was also a 200 with an empty body, where Go sends the
whole payload.
Read Range once with from_utf8_lossy, dropping an empty value, and hand
that one value to the read plan and to both range gates. A replaced byte
never parses, so it is a 416; str::trim trims the same Unicode whitespace
as strings.TrimSpace. A range read from the data file now always answers
itself instead of falling through with an empty needle. The buffered path
answers HEAD before it looks at Range, as Go's writeResponseContent does,
so an EC HEAD with a Range is a 200 with the full length.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: apply Range to chunk manifests and forward raw headers when proxying
A GET of a chunk manifest assembled the object and always answered 200 with
the whole body, ignoring Range. Go serves the expanded manifest through
writeResponseContent, which answers HEAD first and then hands Range to
ProcessRangeRequest: 206 for one range, multipart/byteranges for several,
416 for an unsatisfiable or unparsable one. try_expand_chunk_manifest now
returns the assembled body and headers, and the caller answers through
buffered_response, the same helper the buffered needle path uses.
A proxied read forwarded a request header only if HeaderValue::to_str
succeeded, so a Range with an obs-text byte was dropped and the target
answered 200 with the full body. Go copies every header value as is.
Forward the raw HeaderValue for every header.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: reset Content-Type on range errors as net/http does
Go's http.Error sets text/plain and nosniff unconditionally; keeping the
needle's MIME type on a 416 mislabels the error body. Match it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: count manifest downloads toward the limit
Expanded manifests hard-coded track_download off, so chunked-file reads
bypassed in-flight byte accounting while every other path honored the
download limiter.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A draining filer stayed in the master's lock ring until its process
exited, while gRPC GracefulStop was already refusing new connections. S3
gateways and peer filers kept routing object-write locks and owner-routed
writes to it for the whole graceful-stop window.
On shutdown the filer now sends leave_lock_ring on its open KeepConnected
stream. The master removes it from the lock ring only; it stays a cluster
member so peers keep following its metadata log through the drain. Once
the ring update without it arrives, the filer has already transferred its
locks to the new owners, and it keeps serving through the prior-owner
window (plus a second for peers that apply the update later) before gRPC
and HTTP begin draining. The whole leave is bounded at 10s so slow lock
transfers or a stuck stream cannot hold up the drain. A lone filer, or a
master that ignores the message, falls back to the previous behavior.
* fix(s3): honor object ACLs for anonymous reads
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
* s3: drop unsigned session tokens on deferred anonymous reads
An unsigned GET/HEAD carrying X-Amz-Security-Token is anonymous, not a
session request; strip the token before authentication and authorization
so it cannot influence identity or policy evaluation.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound metadata log flush retries during shutdown
On SIGTERM the filer could hang forever in Shutdown: the final
LocalMetaLogBuffer flush retries appendToFile indefinitely, and with
the master already down each AssignVolume attempt just kept failing.
WaitForShutdown never returned, the interrupt hook never reached
os.Exit, and the half-dead filer kept its ports bound.
Thread a context through appendToFile/assignAndUpload and switch to
a 15s-bounded context once the filer is stopping: the flush abandons
with a log line instead of retrying forever. Normal operation keeps
the unbounded retry so no metadata is dropped while running.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: share one shutdown deadline across all pending meta log flushes
Review feedback on the per-flush timeout: a deadline armed at flush start
could already be expired when shutdown arrived, the first append of a flush
still ran unbounded, each queued window got a fresh budget (16 windows *
15s), and an abandoned window still advanced the flushed watermark as if it
had landed.
Rework to a single shared flush context on the Filer, cancelled once by
Shutdown via AfterFunc. Every append - the in-flight one and every queued
window - observes the same deadline, so the whole drain is bounded at 15s.
A flush that gives up reports its dropped bytes through the new
LogBuffer.NoteFlushDropped, and loopFlush then skips the offset/timestamp
advance and subscriber notifications for that window.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound append attempts and commit uploaded pieces detached
Store operations check then drop request cancellation, so a stalled
backend could still hold flushFn past the shutdown deadline; run each
append attempt on its own goroutine and give up on it at the deadline.
Once a piece is uploaded, commit its entry on a detached context so the
expired deadline cannot strand the chunk.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep shutdown flushes synchronous
Detaching the append attempt let flushFn return while the goroutine
still held the pooled flush buffer and could commit after the metadata
store closed; abandonment is only safe for the cancelable assign/upload
phase, which the shared flush context already bounds.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: read GET/HEAD needles off the store lock, and only once
The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.
The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stream needle chunks without the store lock
StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.
The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.
The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: split get_or_head_handler_inner into phases
get_or_head_handler_inner was a ~650-line function. Its middle resolved
the needle and set five mutable flags (stream_info, can_stream,
can_handle_head_from_meta, can_handle_range_from_source, bypass_cm) that
three if-let reply paths then re-tested, each re-checking stream_info.
It is now a 126-line orchestrator over named phases: reject_read_jwt,
proxy_missing_volume, wait_for_download_slot, parse_read_request,
read_ec_needle / read_volume_needle, etag_and_last_modified,
not_modified_response, read_response_headers, and the reply phases
stream_response, head_from_meta_response, range_from_source_response,
buffered_payload and buffered_response. The read phases return a
ReadPlan whose ReadStrategy enum (Stream, HeadFromMeta, RangeFromSource,
Buffered) carries the NeedleStreamInfo only on the variants that use it,
so the reply is one match instead of three flag checks.
Pure refactor: every status code, header and header order, error text,
metric increment, lock and data-file lease scope, spawn_blocking
boundary and side-effect order is unchanged. Phases that can end the
request return ControlFlow<Response, T>. A Range header that is not
visible ASCII still falls through to the buffered path, as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stop a needle stream once its volume becomes unavailable
Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.
The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: read a non-ASCII or empty Range header as Go does
A Range value with a byte >= 0x80 (obs-text, which hyper accepts) failed
HeaderValue::to_str at both range gates. For a needle served as stored the
handler had already chosen a meta-only read, so it fell through to the
buffered path with no payload and answered 200 with an empty body; a
compressed or EC needle answered 200 with the full body. Go's parseRange
fails on a byte it can neither trim nor parse and answers 416 "invalid
range", and trims Unicode whitespace such as NBSP into a normal 206. An
empty Range value was also a 200 with an empty body, where Go sends the
whole payload.
Read Range once with from_utf8_lossy, dropping an empty value, and hand
that one value to the read plan and to both range gates. A replaced byte
never parses, so it is a 416; str::trim trims the same Unicode whitespace
as strings.TrimSpace. A range read from the data file now always answers
itself instead of falling through with an empty needle. The buffered path
answers HEAD before it looks at Range, as Go's writeResponseContent does,
so an EC HEAD with a Range is a 200 with the full length.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: apply Range to chunk manifests and forward raw headers when proxying
A GET of a chunk manifest assembled the object and always answered 200 with
the whole body, ignoring Range. Go serves the expanded manifest through
writeResponseContent, which answers HEAD first and then hands Range to
ProcessRangeRequest: 206 for one range, multipart/byteranges for several,
416 for an unsatisfiable or unparsable one. try_expand_chunk_manifest now
returns the assembled body and headers, and the caller answers through
buffered_response, the same helper the buffered needle path uses.
A proxied read forwarded a request header only if HeaderValue::to_str
succeeded, so a Range with an obs-text byte was dropped and the target
answered 200 with the full body. Go copies every header value as is.
Forward the raw HeaderValue for every header.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: fetch only the chunks a manifest Range needs, like Go
A ranged GET of a chunk manifest fetched every chunk, assembled the whole
object and then sliced it, so reading a few bytes of a large object cost a
read of all of it, and a 416 still fetched everything. Go serves a manifest
through ChunkedFileReader, which seeks to each range and reads only the
chunks under it.
For a GET with a Range, try_expand_chunk_manifest now parses the ranges
against the manifest size, fetches only the chunks whose declared window
overlaps one of them (none when the reply carries no body), and answers
through handle_range_request_with, the reader-based core that
handle_range_request now wraps, so 206/416/multipart stay one code path.
The reader replays assembly: chunks clamped as before, later chunks over
earlier ones, zeros in gaps. HEAD, no-Range GETs and GETs that crop or
resize an image still assemble the whole object.
A missing chunk outside the requested ranges no longer turns a ranged GET
into a 500, as in Go; a missing chunk inside them still does, before any
headers are sent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: reword a comment codespell flags
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: keep only range-covered bytes of fetched manifest chunks
A ranged GET retained every overlapping chunk's full contents; 1,000
overlapping 8 MiB chunks could pin ~8 GiB for a one-byte response. Clip
each fetched chunk to the bytes the requested ranges can actually read,
preserving the later-chunks-overwrite and zero-fill-gap semantics.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: bucket ranged manifest parts by range
Serving a multipart range scanned every retained part. Bucket the kept
intersections by their range so one range only reads its own parts.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* kafka: honor notification.kafka.event_types
Kafka published every filer event and ignored the filter the webhook notifier already uses.
Co-authored-by: Cursor <cursoragent@cursor.com>
* notification: share event-type classification between queues
Kafka duplicated the webhook's event classification verbatim; move it to
the notification package so the two queues cannot drift. Webhook keeps
its typed eventType wrappers over the shared helpers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-05 21:15:06 +08:00
704 changed files with 58090 additions and 8889 deletions
The same process also runs the master, a volume server, the filer, WebDAV, the Iceberg REST catalog, and the Admin UI. Add `S3_TABLE_BUCKET=warehouse` to also create an Iceberg table bucket, or `warehouse:LANCE` for a Lance one. Drop the AWS keys to run without authentication for development.
Without Admin authentication or mTLS, `weed mini` binds the Admin UI/API and its worker gRPC control plane to loopback rather than `-ip.bind`. Set `WEED_ADMIN_PASSWORD` or configure `https.admin` mTLS to keep the network bind. Remote workers must opt in with `-admin.worker.ip=<address>` and should configure `grpc.admin` mTLS. `-admin.allowInsecureBind` restores the legacy unauthenticated network bind and should only be used on an isolated network.
> macOS: if the binary is quarantined, run `xattr -d com.apple.quarantine ./weed` first.
`weed mini` is auto-tuned for one node and is fine for single-node production, such as an S3 gateway that issues presigned URLs. See [Quick Start with weed mini][WeedMini].
@@ -223,7 +225,7 @@ The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials. [OLake Go][OLake] is a high-performance, open-source EL (Extract–Load) platform that bridges operational databases and open lakehouse storage - writing Apache Iceberg tables or Parquet files on Amazon S3.
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
* IAM at the bucket, namespace, and table level with standard bucket policies, see [S3 Tables Security][S3TablesSecurity].
* A [Hadoop compatible file system][Hadoop] for Spark, Flink, and HBase.
@@ -485,6 +487,7 @@ Your support will be really appreciated by me and other supporters!
@@ -27,6 +27,19 @@ so your deployment will be spread/HA.
* cert config exists and can be enabled, but not been tested, requires cert-manager to be installed.
## Prerequisites
The chart's templates render with Helm 3.16.3 and newer versions tested in CI.
Earlier Helm versions are not covered by this chart's compatibility checks.
When JWT signing is enabled, a live upgrade reuses keys from the existing
security ConfigMap (including the legacy ConfigMap name); a key absent from
that ConfigMap is generated. Plain `helm template` does not read cluster state;
use an install and upgrade against a cluster to check key persistence.
The key reader supports single-line quoted TOML strings in JWT sections,
including indented assignments, quoted key names, and bare or simply quoted
section-name segments. An unsupported value or quoted header fails the upgrade
rather than silently rotating a key.
### Database
leveldb is the default database, this supports multiple filer replicas that will [sync automatically](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Store-Replication), with some [limitations](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Store-Replication#limitation).
@@ -332,8 +345,9 @@ To enable the admin interface, add the following to your values.yaml:
```yaml
admin:
enabled: true
port: 23646
grpcPort: 33646 # For worker connections
port: 12535
grpcPort: 22535 # For worker connections
legacyPort: 23646 # Compat: also serve http + gRPC (+10000) for pre-12535 clients
secret:
adminUser: "admin"
adminPassword: "your-secure-password" # Leave empty to disable auth
@@ -351,7 +365,9 @@ admin:
className: "nginx"
```
The admin interface will be available at `http://<admin-service>:23646` (or via ingress). Workers connect to the admin server via gRPC on port `33646`.
The admin interface will be available at `http://<admin-service>:12535` (or via ingress). Workers connect to the admin server via gRPC on port `22535`.
For backward compatibility with older clients and workers, admin also serves the previous default ports: HTTP on `23646` and gRPC on `33646` (legacyPort + 10000). The Service and NetworkPolicy expose both. Set `admin.legacyPort: 0` to disable.
### Admin Authentication
@@ -423,7 +439,7 @@ worker:
# Auto-discovery only works if the admin is in the same namespace and same Helm release.
# For cross-namespace or separate release scenarios, explicitly set this value.
# Example: If main SeaweedFS is deployed in "production" namespace:
Every component with an `ingress` block (master, volume, filer, s3 and admin) can also be exposed through a [Gateway API](https://gateway-api.sigs.k8s.io/) `HTTPRoute`, for clusters that route through a Gateway instead of an Ingress controller. `<component>.httpRoute` sits next to `<component>.ingress` and is disabled by default. The chart does not create the Gateway, so point `parentRefs` at one that already exists, and the `gateway.networking.k8s.io/v1` CRDs must be installed.
```yaml
s3:
httpRoute:
enabled: true
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
namespace: gateway-system
sectionName: https
hostnames:
- s3.example.com
```
With `rules` left empty the route sends all traffic to the component's own Service and port. A rule may set `matches`, `filters`, `timeouts` and `backendRefs`, and a rule without `backendRefs` still routes to that Service. In all-in-one mode the master, volume, filer and s3 routes target the all-in-one Service. When S3 runs only on the filer (`filer.s3.enabled` without `s3.enabled`), the s3 route uses `filer.s3.port`.
## Network Policies
In a namespace with a default-deny policy the install hangs: the components cannot resolve each other, and the post-install bucket hook waits on the master and filer until it gives up. `networkPolicy.enabled` renders one `NetworkPolicy` per component, selecting its pods by the standard `app.kubernetes.io/{name,instance,component}` labels and admitting traffic from the other pods of the release on the ports that component listens on.
'checked against iam.amazonaws.com (docs list UnrecognizedClientException); '+IAMDOC+'CommonErrors.html#CommonErrors-UnrecognizedClientException',iam({'Action':'ListUsers'},creds=BAD_KEY))
case(2,'IAM ListUsers, wrong secret',(403,'ErrorResponse',None),'undocumented for IAM',
case(3,'S3 request without x-amz-date',(403,'Error','AccessDenied'),'checked against s3.amazonaws.com; '+S3DOC+'API_Error.html#:~:text=Code%3A%20AccessDenied',
S3('GET','/'+B,mutate=no_date))
case(3,'Authorization without Signature=',(400,'Error','AuthorizationHeaderMalformed'),
'checked against s3.amazonaws.com; '+S3DOC+'API_Error.html#:~:text=Code%3A%20AuthorizationHeaderMalformed',S3('GET','/'+B,mutate=drop('Signature')))
case(3,'Authorization without Credential=',(400,'Error','InvalidArgument'),
'checked against s3.amazonaws.com ("Unsupported Authorization Type")',S3('GET','/'+B,mutate=drop('Credential')))
form,exp,chkin(('GET',(400,'Error','InvalidArgument'),'checked against s3.amazonaws.com ("Invalid version id specified"); '),
('HEAD',(400,None,None),'checked against s3.amazonaws.com; '),
print('\n%d cases, %d differ from AWS'%(len(rows),diff))
sys.exit(1ifdiffelse0)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Daniel GarcíaandGitHub
github-actions[bot]
joma1022andGitHub
Peter DoddandGitHub
Robin TweedieandGitHub
scooby87andGitHub
homeofeandGitHub
Bharat KauravandGitHub
Paolo VallettaandGitHub
Younsung LeeandGitHub
UpayanandGitHub
