#11544 made VolumeError::ReadOnly carry the volume id; the batch-write
fallback in write_volume_needles still matched and built it as a unit
variant, breaking the build (E0532/E0308).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
VolumeError::ReadOnly displayed "volume is read-only". Go's store and
volume say "volume %d is read only", and the filer's deletion classifier
requeues a failed delete only when the error contains "is read only".
Against a Rust volume server a BatchDelete on a read-only volume (tier
move, maintenance) was booked as a permanent failure and the chunk was
never deleted.
ReadOnly now carries the volume id and displays Go's text. The text
reaches clients through BatchDelete results, the HTTP write and delete
error bodies, and gRPC statuses; the gRPC code (FailedPrecondition) and
the HTTP/BatchDelete status codes are unchanged.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume: split the write path into reusable steps
do_write_request ran its pre-append checks, the append, the sync
rollback, the index publish and the post-write bookkeeping inline, so a
batched write could only reuse it one needle at a time. Pull the steps
out (check_writable, prepare_write, undo_unsynced_append, publish_write,
finish_write) and the store's volume lookup plus disk-space check
(writable_volume_mut). do_write_request composes them in the same order
with the same early returns; no behaviour change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: group-commit fsync writes in the write queue
The write queue holds one store lock for a batch of up to 128 needles
but wrote them one at a time, so every fsync needle paid its own .dat
sync and its own .idx sync: 2N syncs per batch.
Add Volume::write_needles_grouped, after Go's processBatch. A volume's
entries are split into runs of distinct needle ids (a repeated id starts
a new run, so its dedup and cookie checks see the earlier write). A run
with a durable entry appends everything with append_at_ns chained
through a local, syncs the .dat once, and only then publishes the
entries and syncs the .idx once. A failed .dat sync truncates the .dat
back to the run start (marking the volume unavailable if that fails),
leaves last_append_at_ns and last_modified untouched, and fails every
entry of the run. Runs with no durable entry go through the unchanged
per-needle path. Store::write_volume_needles is the queue's entry point;
the handlers' non-queue path is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: fail closed when a failed append's rollback fails
append_needle discarded the truncate-back result, so a partial write
that could not be rolled back left unindexed bytes on the .dat while
the volume stayed writable; the next append would bury them mid-file,
past the load-time tail check. Route the rollback through
undo_unsynced_append, which marks the volume unavailable when the
truncate fails, so nothing more is appended over an unverified tail.
* volume server: keep a grouped run's I/O error streak from later appends
A synced run stages every append before any entry finishes, so the
success reset in finish_write ran after the failed appends queued
behind the last write to land and erased their media-error streak.
Sent one at a time, those errors would have counted and quarantined
the volume. Skip the reset when an append after the last landed write
added to the streak.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: replay a grouped run's I/O error streak in queue order
Skipping the run's success reset whenever an append after the last
landed write failed kept the errors from before that write as well,
so a run like [EIO, EIO, landed, EIO] reached the quarantine count
that the same writes one at a time (one error) do not. Mark the
streak where each entry is staged and record the run's success at
the last landed write's mark: errors before it are cleared, the ones
after it still count.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: replay a run's I/O error streak in one atomic step
Reads record their outcomes on the tracker without the volume's write
lock, so record_success_at's separate load and store could drop an
error a read counted in between, or restore errors a read had just
cleared. Keep the count and the clear counter in one atomic word and
apply the replay with a single fetch_update.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
In readMode=proxy, proxy_request copied the target's response headers
with HeaderMap::insert, so a header the target sent more than once
(several Set-Cookie, Vary, Link, ...) reached the client with only its
last value. Go's proxyReqToTargetServer adds every value with
w.Header().Add. Append instead of insert; the Server header is still
dropped and status and body handling are unchanged.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: collect EC heartbeats and adjust volume max off the runtime
The volume pass moved to the blocking pool, but the heartbeat task still
called collect_ec_heartbeat and the following EC shard snapshot, and
Store::maybe_adjust_volume_max, directly on its tokio worker.
maybe_adjust_volume_max runs statvfs on every auto-sized disk and stats the .dat
of every writable volume under the store read lock. All of them block the
worker on the node-wide RwLock<Store> whenever a writer holds it or is
queued, and every task sharing that worker stalls with it.
Run the adjustment, on the pulse and after the master changes volume
options, and the EC tick's heartbeat plus shard snapshot through
off_runtime, like the volume pass. apply_master_volume_options now only
reports whether the options changed; the loop adjusts off the runtime.
What is collected and sent, and in what order, is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: keep EC shard deltas out of the volume heartbeat's snapshot
EC shard deltas are the diff between the store's shards and last_ec_shards,
taken when volume_state_notify fires. But the volume tick and the
options-changed heartbeat re-took last_ec_shards from the store too, and a
volume heartbeat carries no shard list: a mount or unmount that landed while
the notify was pending or the volume pass was collecting was absorbed into
the baseline and never sent. The master only learned of it at the next EC
tick, 17 pulses later. The EC tick likewise built its heartbeat and its
baseline under two separate store guards, so a mount between them was lost
the same way.
A volume heartbeat now only takes out of the baseline the expired EC shards
it reports deleted itself, so the next delta does not repeat them. The EC
tick, and the initial EC heartbeat, build the full list and the baseline
under one read guard, still on the blocking pool.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: take the heartbeat's remaining store reads off the runtime
The heartbeat's volume and EC passes and the volume max adjustment already
run on the blocking pool, but several reads of the node-wide RwLock<Store>
were still taken directly on the heartbeat's tokio worker: the digest
report reset before the first heartbeat, the duplicate-UUID directory
lookup and the volume options a master response carries, the EC shard list
a state notification is diffed against, and the deregistration heartbeat
sent on stop and shutdown. The lock is writer-preferring, so with a writer
holding or queued for it each of these parks the worker, and every task
sharing that worker stalls with it.
Run each through off_runtime, which now takes a closure so a pass can carry
what it needs from the master's response. The notify branch's volume
snapshot and EC read become one blocking pass, still under two guards in
the same order. What is collected and sent, and in what order, is
unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume server: refuse a tier move while compacting, and a commit once tiered
A tier move to remote and a vacuum compaction of the same volume could
interleave and leave the volume unreadable:
- A compaction committing while the upload ran swapped .dat/.idx under
the transfer, which reopens the .dat by path per part. The move then
published an object holding the old (or a mixed) layout against the
compacted .idx, and with keep_local_dat_file=false deleted the only
compacted .dat.
- A tier move finishing while the compaction copy ran (or between the
copy and the commit) let the commit swap in the compacted .idx while
the reload served the pre-compaction remote object through it.
The tier move now refuses to start while the volume is compacting, and
re-checks the compaction revision under the store write lock before it
records the remote file; on a mismatch it deletes the uploaded object
and fails with FailedPrecondition, leaving the volume local. Committing
a compaction on a volume that has a remote file is refused and its
.cpd/.cpx removed, since the reload would read the remote object
through the compacted index.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: abort a tier move whose volume was replaced or removed
The tier-up bookkeeping looked the volume up by id only and compared the
compaction revision. A delete and re-create of the same id during the upload
yields a fresh volume at the same revision, so the move recorded the old
volume's object on the new one and, without keep_local_dat_file, removed the
new .dat. An unmounted volume was skipped and the move reported success,
leaving the uploaded object referenced by nothing.
Capture the volume instance (its data-file access control Arc, as the scan
and read plans do) with the revision, and require both under the store write
lock. A replaced volume fails with FailedPrecondition, a missing one with
NotFound; either way nothing is recorded and the object is deleted after the
lock is released. Go fails in both cases because deleting or unmounting closes
the descriptor its copy reads.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
EcVolumeShard::read_at and the scrub plan's EcLocalShard::read_at were a
single pread/seek_read. That may legally return fewer bytes than asked
mid-file (FUSE/NFS/CIFS mounts, a signal, very large requests), and an
Interrupted error was not retried. Callers treat a short count as end of
file or corruption: verify_ec_shards compared a zero tail and reported a
parity mismatch, local scrub reported a broken shard, VolumeEcShardRead
ended the stream early, and decode/rebuild/local needle reads failed.
Add storage::io::read_full_at, which loops until the buffer is full or a
read returns 0 and retries Interrupted, so a short count means EOF. Route
both shard read_at methods through it, replace the encoder's private
read_at_most with it, and reuse it for the Windows read_exact_at loop.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: ec.decode reads the .ecx from the index dir it was copied to
VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so
with a split data/index dir the decode target has no .ecx beside its
shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but
built the .idx from the data dir, failing with NotFound after the .dat
was already published. It now reads .ecx/.ecj from where the EC volume
opened them and writes the .idx beside the .dat, where Go leaves it.
The live-entry check and the .dat size also ignored deletions recorded
only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first:
a fully deleted volume was decoded instead of reported as having no live
entries, and deleted tail needles were copied into the .dat. Both now
treat journaled ids as deleted, without rewriting the sealed .ecx.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: ec.decode keeps the decoded volume writable and reads every .ecj
The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after
the .dat was cut short before it, so the mount saw a row past EOF and
marked the decoded volume read-only. Rows of deleted needles the .dat no
longer holds are now dropped, and each journaled needle still in the .dat
gets one tombstone instead of one per journal entry.
VolumeEcShardsCopy appends journals collected from other holders into
the idx dir, but the decode read only the .ecj beside the .ecx, which
sits in the data dir when this server generated the shards. It now
reads both, once, in bounded chunks via the loader EcVolume uses.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: test ec.decode drops a sealed .ecx tail tombstone
Covers the other half of the rule added in the previous commit: a tail
needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from
the .dat, and its row must not reach the rebuilt .idx either.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: VolumeNeedleStatus reads remote EC shards and reports deleted needles like Go
For an EC volume the handler read only locally mounted shards, so a node
that did not hold the shard with the needle's bytes answered Internal
"ec shard N not available locally". Go's ReadEcShardNeedle fetches the
interval from a peer or reconstructs it. It also mapped every regular
volume read error, including a tombstone, to NotFound "needle not found",
which fs.verify treats as a missing needle; Go returns ErrorDeleted as a
plain error ("already deleted"), which fs.verify skips.
The EC branch now drops the store guard and uses the distributed EC read
the HTTP GET path uses. Errors map like Go: needle absent -> NotFound
"needle not found <decimal id>", tombstoned (regular or EC .ecx/.ecj) ->
Unknown "already deleted", anything else -> Unknown with the error text.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: tell EC deletions and vanished volumes apart in VolumeNeedleStatus
The distributed EC reader returned Ok(None) for an absent needle, a needle
a peer reported deleted, and a volume unmounted after the handler's own
existence check. VolumeNeedleStatus answered all three NotFound "needle not
found", which fs.verify -pruneEntries counts as lost data. A reported
deletion was also lost when an earlier interval failed.
The reader now says why it has no needle (EcMiss: NotFound, Deleted,
VolumeNotFound), classifying the local tombstone itself and letting a
reported deletion outrank other interval errors, as Go's ReadEcShardNeedle
does. VolumeNeedleStatus maps Deleted to Unknown "already deleted" and
VolumeNotFound to "volume not found", and drops its separate EC pre-check.
read_ec_shard_needle_distributed keeps its Ok(None) for every miss, so the
other callers are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The volume pass moved to the blocking pool, but the heartbeat task still
called collect_ec_heartbeat and the following EC shard snapshot, and
Store::maybe_adjust_volume_max, directly on its tokio worker.
maybe_adjust_volume_max runs statvfs on every auto-sized disk and stats the .dat
of every writable volume under the store read lock. All of them block the
worker on the node-wide RwLock<Store> whenever a writer holds it or is
queued, and every task sharing that worker stalls with it.
Run the adjustment, on the pulse and after the master changes volume
options, and the EC tick's heartbeat plus shard snapshot through
off_runtime, like the volume pass. apply_master_volume_options now only
reports whether the options changed; the loop adjusts off the runtime.
What is collected and sent, and in what order, is unchanged.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mount_and_reply ignored the Result of store.unmount_volume when a copied
replica failed record count validation, tripping unused_must_use. The Err
branch is unreachable there: the volume was mounted under the same store
write guard, mount_volume refuses an already loaded vid so it is a fresh
Volume with is_compacting false, and a compaction claim needs &mut Volume,
i.e. the store lock. Ignore the result explicitly with a one-line reason.
Store::delete_expired_ec_volumes and Store::remove_ec_volume are called
only from test modules (the heartbeat uses the split
find_expired_ec_volumes / remove_expired_ec_volumes halves), so mark them
#[cfg(test)].
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GetObjectAttributes accepted the Checksum attribute but never filled it
in, as its comment said SeaweedFS did not store S3 checksums. PutObject
and CompleteMultipartUpload store them now, and HeadObject returns them.
Fill in Checksum from the same entry fields, with the ChecksumType and
ChecksumCRC64NVME members the response did not have.
Also run ceph/s3-tests' test_get_checksum_object_attributes in CI.
* s3api: do not store aws-chunked in an object's Content-Encoding
aws-chunked in Content-Encoding names the SigV4 streaming framing of
the request body, which the gateway decodes on upload. PutObject and
CreateMultipartUpload stored the header as sent, so an object uploaded
with "gzip, aws-chunked" was served with that Content-Encoding, and one
uploaded with "aws-chunked" alone was served as aws-chunked. S3 drops
aws-chunked and keeps the other encodings.
Also run ceph/s3-tests' test_object_content_encoding_aws_chunked in CI.
* s3api: read every Content-Encoding field, and drop aws-chunked on copy
A client can send aws-chunked and the object's own encoding as separate
Content-Encoding fields. Only the first was read, so "aws-chunked"
followed by "gzip" left the object without its gzip. Combine all the
fields before dropping aws-chunked. CopyObject with the REPLACE
directive stored the requested Content-Encoding as sent: drop
aws-chunked there too.
A replica's read-only flag in the volume layout only moved on registration
and on volume.mark. A change that arrived in the regular heartbeat updated
the node's record, which the writable list follows, but not the layout
flag, which the vacuum sweep reads. So the sweep kept trying volumes on a
disk that had gone read-only while the server ran, and after a restart it
skipped volumes that had since become writable again until the next
restart (issue #11516).
Apply the reported state to the flag for every changed volume. Only the
flag: the writable list stays with EnsureCorrectWritables and its
capacity guards.
* shell: say which read-only volumes volume.vacuum leaves alone
volume.vacuum without -volumeId runs the same sweep as the automatic
vacuum, which skips read-only volumes, and the master's response carries
no result. An operator whose disk filled up runs the command, sees it
return, and watches nothing change (issue #11516).
Before issuing the request, list the read-only volumes whose garbage is at
or above the threshold and point at -volumeId, which is the explicit path
PR #9861 opened for them. The help text says the same.
* shell: volume.vacuum hint survives a failed listing and looks at every replica
Review follow-ups: a failed topology listing no longer stops a sweep
without -volumeId, it only drops the hint; a volume counts as read-only
when any replica is, with the garbage ratio taken from the replica that
reports the most, which is what the sweep itself does; a converted index
that reports deletes without sizes is listed rather than hidden; and the
threshold is printed as given instead of rounded to two decimals.
* shell: do not guess a garbage ratio for a converted index
The master cannot compute one for a volume that reports deletes without
their sizes, and a guess of 1 would send the operator to -volumeId for a
volume the server may decline at that threshold. Leave it out and say so.
* s3/iam: manage roles through the IAM API, with an opt-in persistent role store
Roles could only come from the IAM config file: the S3 server pinned the
role store to memory and the embedded IAM API had no role actions, so a
role could not be created, retrusted or revoked without editing the file
and restarting every gateway.
Role store
- Read the `roleStore` key (the IAMConfig field already existed). With an
IAM config file the default stays memory; with none it is the filer, as
for OIDC providers, so zero-config clusters keep runtime-created roles.
- Roles from the IAM config file never go into a persistent role store,
which outlives the file and may be shared by S3 servers with different
files. They are served from memory beneath the store, as OIDC providers
are: a stored role of the same name takes precedence, and deleting it
restores the file's. A config-file role cannot be changed or deleted
through the API (UnmodifiableEntity), and removing one from the file
removes it at the next start. An in-memory store holds them as records,
as before. They have no creation time, so CreateDate is omitted rather
than reporting when this server started. SetRoleStore installs a store
the same way, so a store set after startup keeps the config-file roles,
as SetOIDCProviderStore does for providers.
- Watch /etc/iam/roles and drop the cached role definitions on change. The
cached filer store otherwise serves a peer's stale role for up to its 5m
TTL, which keeps a revoked trust policy in force on the other gateways.
- Role stores wrap ErrRoleNotFound for a missing role; the filer store
used to report any failed lookup as "role not found". CreateRole proceeds
only on a confirmed absence, so an unreadable store cannot let it write
over an existing role.
IAM actions
- CreateRole, GetRole, ListRoles, DeleteRole, UpdateAssumeRolePolicy,
AttachRolePolicy, DetachRolePolicy, ListAttachedRolePolicies. The reads
are allowed in read-only mode.
- A role defined in the config file is reloaded from it at every start, so
changing or deleting it through the API is refused (UnmodifiableEntity)
rather than silently reverted.
- DeleteRole with policies attached is refused (DeleteConflict), as on AWS.
- Role names follow AWS's rules ([\w+=,.@-]{1,64}); a role is stored as
<name>.json in the filer, so this also keeps a name from leaving the role
store's directory. At most 10 managed policies per role (AWS's default
quota; MaxManagedPoliciesPerUser is 10 too), LimitExceeded beyond.
- DeletePolicy is refused (DeleteConflict) while a role attaches the
policy, as it already is for users and groups: roles attach policies by
name, so a policy created later under the deleted one's name would
otherwise take effect on the role.
- Role paths other than "/" and role tags are not stored, so they are
refused rather than dropped.
Role IDs and sessions
- Roles get a unique RoleId when first stored (random, AWS AROA form),
kept across updates; a config-file role gets a stable ID derived from its
name, since it is created again at every start.
- Sessions issued through AssumeRoleWithWebIdentity, AssumeRoleWithCredentials
and AssumeRole carry the role's ID (claim "rid"), and a request under a role
whose current ID differs is denied. Resolving a session's policies by role
name let a session outlive its role: once a role was deleted, a role later
created under the same name — with a different trust policy and different
policies — revived every unexpired session of the old one with the new
role's permissions. Sessions issued before this change carry no ID and are
unaffected until they expire.
Integration test (test/s3/iam, run with `make start-services`):
TestWebIdentityWithProviderAndRoleManagedThroughIAMAPI configures an OIDC
provider, a managed policy and a role entirely through the IAM API against a
JWKS served by the test, then checks the trusted subject gets credentials
scoped to the attached policy; another subject, a token signed by another
key, an unsigned token and a token for another audience are refused; and UpdateAssumeRolePolicy moves the
trust at once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: bind every role session to its role, and change roles atomically
Review follow-ups.
Session binding
- The role-ID check ran only when a session carried no policy names, and
AssumeRole embeds the role's attached policies, so those sessions kept
their permissions after the role was deleted or recreated. The check
now runs for every session carrying a role ID, before policy selection.
- A named role that cannot be resolved at issuance gets no session,
instead of one with no role ID (which nothing binds).
- A config-file role's ID is derived from its name and trust policy, not
the name alone: a different role put in the file under the same name
gets a new ID, while an unchanged role keeps its sessions across restarts.
Role writes
- RoleStore gains UpdateRole, a read-modify-write that lands only if the
role is unchanged since the read, and otherwise re-reads and retries. The
filer store uses the filer's write conditions (IF_NOT_EXISTS for a new
role, IF_ENTRY_EQUAL otherwise). CreateRole, UpdateAssumeRolePolicy and
Attach/DetachRolePolicy all go through it, so two gateways no longer
overwrite each other's changes, a change racing a delete no longer
writes the role back, and of two concurrent creates one gets
EntityAlreadyExists.
- The filer store's ListRoles pages past 1,000 entries and fails on a
broken stream instead of returning what arrived, so DeletePolicy's
attachment check sees every role. ListRoles skips a role deleted between
listing and reading it.
- CreateRole validates first; a failed write is ServiceFailure, not
InvalidInput. Any Tags.* parameter is refused, not only the first key.
- ExecuteAction's skipPersist covers the S3ApiConfiguration only; the
comment now says so. Role and OIDC provider actions write their own stores.
Each fix has a test that fails without it. Against a real filer with two
gateways, concurrent AttachRolePolicy calls lost 1-4 of 8 attachments per
run before this change and none after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: one role snapshot per decision; DeleteRole is atomic; watch a custom role store path
Review follow-ups.
- Authorization evaluates the policies of the role definition the session's
binding was checked against, instead of reading the role again: a role
replaced in between cannot lend a session its policies.
- AssumeRole and AssumeRoleWithLDAPIdentity issue the session from the
definition whose trust admits the caller (IAMManager.ResolveRoleForPrincipal),
and take its ID, duration cap and embedded policies from that same
definition. A role replaced after the caller's trust check by one that does
not trust the caller now yields AccessDenied, not a session bound to the
replacement.
- A RoleUpdate that returns nil deletes the role, on the same condition as a
write: the filer store deletes with ObjectTransaction on IF_ENTRY_EQUAL,
routed and locked like the conditional CreateEntry. DeleteRole decides
against the role it deletes, so a policy attached meanwhile on another
server is a DeleteConflict, and a delete never removes a role written
after its check.
- S3 servers watch the role store's configured basePath, not only
/etc/iam/roles, so a custom path also drops peers' cached roles on change.
Each has a test that fails without it. Live against a real filer: DeleteRole
refuses while a policy is attached and removes the entry once detached; all
test/s3/iam CI stages pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: state which roles DeletePolicy's attachment check can see
RolesAttachingPolicy sees the stored roles and this server's config-file
roles. A role defined only in another server's IAM config file is invisible
to it, so a config-file role that attaches a managed policy is protected
only on the servers whose file defines it. The doc comment now says so and
how to avoid it: keep such roles in every server's file, or attach only
config-file policies to config-file roles.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: note that a role store set after startup is not watched for peer changes
S3 servers build their metadata watch list once, at startup, from the role
store installed then. SetRoleStore's doc now says that a filer-backed store
installed later with a different basePath is not watched, so peers' changes
to it reach this server's cached roles only when the cache expires.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: retry a failed OIDC provider refresh until the store answers
RefreshOIDCProvidersFromStore reports a failure and nothing retries it. Its
callers can't: a metadata-subscription event reports each change once, so a
refresh that found the filer unreachable on it (the filer restarting, say)
left a peer's new provider untrusted, or a deleted one trusted, until some
unrelated later change. The refresh after a local IAM API mutation has the
same shape. Only the startup load retried.
A failed refresh now retries in the background with the startup load's
backoff until the store answers. At most one retry runs, however many
refreshes fail meanwhile, and installing another store cancels it. The
startup load uses the same path instead of its own.
Seen on a SeaweedFS operator cluster whose filer restarted while an
S3OIDCProvider was created: the gateway logged "OIDC provider refresh after
/etc/iam/oidc-providers change failed: ... fail to dial". The operator's
periodic re-apply happened to recover it; an IAM API client would not.
* s3/iam: never retry or apply a superseded OIDC provider store, and never drop a failure during a retry
Review of the retry (#11521) found two ways to lose the state it protects.
A refresh of store A that failed as store B was installed could start a
retry for A after B's install had cancelled retries. Nothing cancelled it,
and when A answered it replaced B's providers in STS. The installed store
now changes under the retry lock, a store that is no longer current gets no
retry, and a snapshot of a replaced store is never handed to STS, even when
the refresh listed it just before the swap.
A refresh that failed while a retry ran was dropped by the at-most-one
guard, though the retry might already have listed an older snapshot, so the
change the failed refresh would have loaded stayed unloaded. The retry now
runs once more after its success when a failure arrived meanwhile.
Each has a test that fails without its guard.
2026-09-30 17:33:40 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage/topology: keep disk-full read-only volumes vacuumable
The vacuum sweep skipped every read-only replica, so a volume that went
read-only because its disk filled could never reclaim its garbage — the
exact situation compaction exists for. The volume server now reports
disk_space_low in VacuumVolumeCheckResponse, and the sweep skips a
read-only replica only when the flag is clear. An explicit volumeId
vacuum is unaffected: it already bypassed the read-only rule.
The field takes number 4: 2 and 3 are downstream-allocated for tombstone
retention, keeping the wire merge clean.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: measure vacuum free space against live bytes
The pre-compaction space check required the current .dat + .idx size
free, which includes the garbage being reclaimed — on a nearly full disk
that estimate can never fit, so the volume stayed garbage-bound forever.
Measure against the estimated compacted output instead: superblock plus
live index entries plus live content bytes, with the existing ten
percent buffer unchanged. Mirrors the same check in the Rust volume
server.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: count per-needle framing in the compacted-size estimate
The live-bytes estimate covered each live needle's content and index
entry but not its .dat framing (header, checksum, timestamp, padding —
~32 bytes on version 3). For small-needle volumes that is more than the
10% headroom, so a disk with space between the estimate and the real
output still ran out mid-compaction. Rust side mirrors the same formula.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: report disk_space_low only when it is the sole read-only cause
Review feedback (ihnokim, greptile, devin): a volume read-only for low
disk space AND an operator mark or I/O quarantine was still eligible for
the automatic sweep, rewriting a copy meant to stay protected. The flag
now reports only the benign sole-cause case in both servers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: fail closed when the read-only lookup misses in the sweep
A heartbeat can drop the volume from the DataNode cache between the
location-list copy and VacuumVolumeCheck; a lookup error previously
skipped the read-only check entirely. Review feedback (coderabbit).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CopyObject responses carried LastModified values such as
"2026-09-29T20:30:04.56Z": trailing zeros of the fractional seconds were
trimmed, and a whole-second value had no fraction at all. AWS S3 always
writes exactly three digits ("...04.560Z"), and clients that parse with a
fixed-width pattern reject anything else. minio-java 8.6.0
(yyyy-MM-dd'T'HH:mm:ss.SSS'Z') throws DateTimeParseException, so roughly
one CopyObject in ten fails on the client even though the copy succeeded.
Two causes:
- xsdDateTime marshalled with "2006-01-02T15:04:05.999999999", which
drops trailing zeros. It now writes UTC with ".000Z".
- CopyObjectResult.MarshalXML had a pointer receiver, but the handlers
pass the result by value, so encoding/xml never called it and fell back
to time.Time's RFC 3339 encoding. It now has a value receiver.
CopyPartResult had no custom marshaller at all; it now uses xsdDateTime.
Follow-up to #8394 / #8398, which truncated these timestamps to
milliseconds but kept the trimmed format.
* filer: identify remote-mounted entries safe to drop under disk pressure
ListEvictableRemoteEntries walks every mounted directory directly on the
filer store (no lazy remote listing) and returns entries that hold local
chunks fully synchronized with remote, ordered oldest-cached first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: evict remote-cached chunks oldest-first and vacuum the garbage
uncacheRemoteEntry applies the same transition remote.uncache does -
cleared chunks plus a reset LastLocalSyncTsNs under the entry path lock -
and evictRemoteCachedEntries serializes passes over all mounts until a
byte target is met. Aged victims are preferred; a second pass accepts any
synchronized cached entry when aged ones cannot cover the request, since
a failed read is worse than a dropped hot object.
Cleared chunks only become disk space after compaction, so
reclaimRemoteCacheSpace pairs each pass with a rate-limited VacuumVolume
call that also picks up orphaned partial fills.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: trigger remote cache eviction under storage pressure
A periodic check (30s) reads disk usage from master topology and evicts
remote-mounted cached chunks once any disk crosses
-filer.remoteCacheEvictThreshold (default 0.9; 0 disables), with a vacuum
pass to reclaim the tombstoned needles.
The cold-read cache path also kicks the same reclaim when a fill fails on
exhausted volumes - the request still falls back to streaming from the
remote, but the cache stops being permanently wedged full.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: flush deletion queue before remote cache vacuum
Vacuum ran immediately after eviction while evicted file IDs still sat
in the asynchronous deletion queue, so compaction saw no garbage and the
cache stayed wedged. Flush the queue synchronously first and shorten the
vacuum cooldown so sustained pressure does not wait five minutes between
reclaim passes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover remote cache eviction under capacity pressure
Unit tests pin the eligibility filter and oldest-first ordering; the
integration test runs a constrained two-node setup that saturates the
cache, verifies the oldest synced entry is evicted and vacuumed, and
that a later read re-caches it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: coalesce remote cache reclaim passes
A failed cache fill used to queue behind any in-flight eviction,
stacking full mount traversals during a write-failure storm. Skip the
pass when one is already running; the caller falls back to streaming
from remote regardless.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: stop the remote cache janitor on shutdown
The eviction ticker kept running after Shutdown closed the metadata
store and could traverse a closed store. Give the janitor a context
cancelled from Shutdown and propagate it into its master RPCs and
traversals.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: vacuum only tombstoned volumes and retry deferred passes
VacuumVolume with no volume id swept every collection, compacting
volumes unrelated to the cache fill that failed. Now the reclaim path
collects the vids of file ids actually flushed from the deletion queue
and compacts only those. Vids that land inside the vacuum cooldown stay
in a pending set the janitor retries on each tick, so chunks evicted
just after a sweep are not stranded until the next pressure event.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: count only pressured disks when evicting remote cache
The janitor measured the largest excess on one disk but let bytes on
healthy disks satisfy the reclaim target. Split the topology disk view
per physical disk and count only chunk bytes whose volumes sit on an
over-threshold disk; entries contributing nothing there are skipped.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: compare remote cache sync time at nanosecond precision
Second-precision mtime comparisons let a local write in the same second
as the last sync still qualify as evictable, discarding unsynced
changes. Compare LastLocalSyncTsNs against full-precision mtime
(mtime_ns round-trips through the entry codec), and apply the same fix
to remote.uncache's inline check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: invalidate remote sync stamp on local content change
A local overwrite that keeps the remote entry's LastLocalSyncTsNs looks
evictable even though the remote copy no longer matches, and some write
paths stamp mtime at second precision so a timestamp comparison cannot
catch it. UpdateEntry now clears the stamp when chunks change without a
fresh stamp, leaving replicated updates authoritative.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound remote cache master rpcs and vacuum all evicted garbage
VolumeList and VacuumVolume now run under a 30s context so a stalled
master cannot wedge the eviction janitor. The targeted vacuum drops the
garbage threshold so volumes with under 10% deleted bytes still compact.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: tolerate straggler fills in remote cache eviction test
Detached fills from the concurrent wave keep racing the final checks:
live chunks legitimately fill both volumes, and a re-cached object can
be evicted again before its commit is observed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: start remote cache eviction loop after filer init
The janitor's first tick dereferences fs.filer; starting the goroutine
before NewFiler assigns it could panic when startup exceeds an interval.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep remote cache vacuum intent across retries
Evicted entries now record their chunk volumes for vacuum directly, so
the intent survives whoever consumes the shared deletion queue first.
A pending volume keeps several vacuum attempts so tombstones that land
late are still compacted, and the janitor retries pending volumes under
the reclaim mutex instead of flushing unrelated deletes every tick.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound each remote cache vacuum request independently
A shared 30s deadline across pending volumes let one slow compaction
cancel the rest. Each VacuumVolume now gets its own context, and pending
volumes keep more attempts since the master reports request acceptance
rather than compaction.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: tighten remote cache reclamation bound
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: treat chunk timestamp changes as content changes
chunksEqual now also compares ModifiedTsNs so an update that rewrites a
chunk record still invalidates the remote sync stamp.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: run remote cache queue flush under the reclaim context
BatchDelete for flushed file ids now uses the caller's context instead of
context.Background(), so a reclaim pass bounded by shutdown or timeout
stops its deletes too. Other callers keep their existing behavior.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: scope remote cache vacuum to evicted volumes
The flush no longer feeds the shared deletion queue's ids into the
pending set — only evicted chunks' volumes are tracked, so ordinary
deletions no longer pick up repeated vacuum attempts. The flush also
runs under a shutdown-immune bounded context and is skipped when no
volume is pending.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: retry remote cache vacuums even after unmount
Pending volumes were only retried while a remote mount existed; removing
the last mount skipped every later pass and left evicted bytes allocated.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: reclaim partial cache fills that run out of capacity
A fill that fails midway queues its written chunks for deletion, but
when no entries remain evictable the reclaim pass found no pending
volumes and skipped the flush and vacuum entirely, leaving the partial
garbage to the slow periodic vacuum while the disk stayed full. Mark
the failed fill's chunk volumes pending so the pass tombstones and
compacts them even when nothing was evicted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add ppc64le and s390x to the normal and large_disk release image
matrices, next to arm/v7 and 386, and to the manual container_latest
fallback so that rebuilding `latest` does not drop them from the
manifest.
The Go binary is cross-compiled without cgo on the build host, and
Dockerfile.go_build already falls back to empty Rust volume server and
worker placeholders on platforms without pre-built Rust binaries. Only
the small final-stage RUN steps run emulated, so these two platforms use
QEMU on ubuntu-latest like arm/v7.
Refs #5558
2026-09-29 21:04:19 +08:00
Khris RichardsonGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Claude Opus 5.5Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3/iam: persist IAM-managed OIDC providers in the filer, and trust them after a restart
The S3 server's IAM config loader never read the documented
`oidcProviderStore` key, so the OIDC provider store was always in memory:
a provider created with CreateOpenIDConnectProvider lived in one gateway's
process, was lost on restart, and was never seen by peers. The
/etc/iam/oidc-providers metadata subscription refreshed from that empty
in-memory store.
- Read `oidcProviderStore` and pass it to the IAM manager. With an IAM
config file the default stays memory. With no config file (zero-config
IAM, as `weed filer -s3` and operator-managed clusters run) it defaults
to the filer: there is nothing static to shadow, and providers created at
runtime otherwise vanish on restart.
- With a store that outlives the process, load the STS runtime view from it
at startup, so providers created on an earlier boot or on a peer are
trusted without waiting for the next mutation.
- If the store cannot be read at startup (a filer not up yet), the load is
retried in the background with backoff until it succeeds: the metadata
subscription reports only later changes, so providers already stored would
otherwise stay unknown to STS until one of them changed.
- Mark records mirrored from STS.Providers as `source: static-config`, and
at startup delete such records whose provider has left the config, so
removing a provider from the config file still revokes it. Records created
through the IAM API are never pruned.
- The filer store reported every failed lookup, an unreachable filer
included, as ErrOIDCProviderNotFound, which CreateOIDCProvider reads as
"free to create". Only a confirmed absence is now not-found.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: keep config-file OIDC providers out of a persistent store
Review of the previous commit found that mirroring the IAM config file's
providers into a persistent store, and pruning them when they leave the
file, breaks as soon as S3 servers share a filer:
- a server prunes stored config-file providers its own file does not list,
including ones a peer's file still defines (a zero-config server prunes
them all);
- mirroring overwrites an API-created provider with the same ARN and marks
it config-owned, so a later prune deletes it;
- a failed mirror write or a failed prune leaves a stale record trusted;
- a mirrored record is loaded into STS at startup as an IAM-managed provider
and shadows the config-file provider, dropping the settings a record does
not carry (jwksUri, roleMapping, policyClaim, ...).
A persistent store now never receives the config file's providers. STS keeps
serving them from its static configuration, as it always has; the IAM API
lists and returns them from memory, refuses to change or delete them
(UnmodifiableEntity; change them in the file) and to create another provider
with their ARN (EntityAlreadyExists). The store holds only providers created
through the IAM API, and those are what startup loads into STS. There is
nothing to prune, so the source marker is gone. An in-memory store keeps its
behaviour: the config file's providers are records in it, as before.
buildOIDCProviderFromRecord also carries PolicyClaim and
AllowedPrincipalTagKeys now; they were dropped whenever an API-created
provider was loaded into STS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: send UnmodifiableEntity as a 400, not an internal error
The IAM API's error writer had no case for UnmodifiableEntity, which the
previous commit returns for a change to a config-file provider, so it went
out as a 500 ServiceFailure that clients retry. AWS sends it as a 400.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: document stored-over-config precedence, drop invented CreateDate, cancel superseded retries
Follow-ups from review of b881982d2:
- A provider stored under the same ARN as a config-file provider takes
precedence in the IAM API, matching STS, which already prefers
IAM-managed providers so that an API call can shadow a bootstrap entry.
Deleting the stored provider brings the config-file one back. This was
already the behaviour; it is now documented and tested.
- A config-file provider no longer reports its server's start time as
CreateDate, which changed on every restart; GetOpenIDConnectProvider now
omits the date for it. An in-memory store still stamps its copies at load,
as before.
- The startup retry runs under a cancellable context, is cancelled when
another store is installed, and retries the store it was started for
rather than reading the manager's field, so replacing the store neither
leaves the old retry running nor races with it (go test -race).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: serialize OIDC provider refreshes so an older snapshot cannot restore a deleted provider
Refreshes run concurrently: after an IAM API change, on a peer's change
and in the startup retry. Each lists the store and then hands STS the
result, so a refresh that listed before a DeleteOIDCProvider could finish
after that call's own refresh and keep the deleted provider trusted until
the next change. Refreshes now hold a lock from the read to the hand-off,
and a startup retry cancelled by installing another store drops its
snapshot instead of applying it.
The retry-cancellation test waits for the retry by polling instead of a
fixed sleep.
* s3/iam: route SetOIDCProviderStore through installOIDCProviderStore
A store installed after Initialize skipped the static-provider overlay
and startup hydration: config-file providers disappeared from the IAM
API, ErrOIDCProviderStatic no longer protected them, and stored
providers were never trusted until the next mutation or peer event.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-29 11:51:39 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: evaluate trust policies deny-by-default
EvaluateTrustPolicy seeded its result with the engine's DefaultEffect,
so a non-matching trust-policy statement set still resolved to Allow
when the IAM config sets policy.defaultEffect=Allow. A caller holding
a validly signed token from a registered provider could then assume a
role its trust policy does not admit.
Trust policies now start from implicit deny, matching AWS semantics and
the pre-d751623 behavior of evaluateTrustPolicy; DefaultEffect still
governs identity-policy evaluation.
Upgrade note: deployments on defaultEffect=Allow whose trust policies do
not match their callers will see those assumptions refused.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: cover trust policy implicit deny under DefaultEffect=Allow
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-29 11:36:01 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The stored entry came through FindEntry, which restores chunk file ids
from their fid form, while an expected entry built from a metadata-log
event still carries the serialized form (file_id moved into fid). The
proto.Equal saw file_id "" against the restored id and refused every
stamp, so remote.sync re-uploaded each entry and the RemoteEntry stamp
never landed.
Clone both sides and run BeforeEntrySerialization before comparing, so
chunks match on their fid and the file_id spelling is ignored; the stored
entry and the request's ExpectedEntry are left untouched.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: synchronize stream pins and release them on transitions
Guard chunkStream.cacher with the ReaderCache lock everywhere: mount
sections share one ChunkReadAt across concurrent reads, and unsynchronized
release could double-unpin. Reads served from the chunk cache now detach
the stream's pin instead of retaining the previous chunk. Eviction prefers
unpinned downloaders so a pinned buffer is not dropped mid-stream. A new
ReleaseStream lets callers drop their pin without destroying the shared
cache; S3 and WebDAV readers use it. lastChunkFid becomes atomic since
concurrent mount reads can update it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep eviction bounded when every downloader is pinned
Both eviction paths still fall back to a pinned victim when no unpinned
one exists, so abandoned stream pins cannot bypass the downloader limit
or stall the memory budget. Budget eviction also rechecks the pin under
the ReaderCache lock at removal time: a stream that pinned the selected
victim in between keeps it mapped and the selection retries.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: restore budget bookkeeping when a victim gets pinned mid-eviction
removeUnpinned losing the pin race left the victim out of the idle list
while still holding its reservation, making it unevictable even as the
pinned fallback. Push it back when the reservation is still live.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The ReaderCache is shared by all streams of a process (every S3 GET, for
instance), but a ChunkReadAt released chunks as if it owned them:
- moving on to the next chunk called UnCache on the previous one,
destroying the buffer even when other streams were still inside it;
- since #11384 a buffer is dropped once any reader has consumed it to the
end and no read call is in flight. Streams copy out in slices (256 KiB
in the S3 gateway), so between two calls a slower stream is not
attached and loses the buffer to a faster one.
Either way the slower stream refetches the whole chunk from the volume
servers. With many clients downloading the same popular object at once,
each chunk is fetched over and over; in production we saw the S3 gateway
pull ~10 Gbit/s from volume servers while serving ~1 Gbit/s to clients.
A ChunkReadAt now pins the chunk it is positioned in. The pin is taken
and released only under the ReaderCache lock, since concurrent ReadAt
calls on one ChunkReadAt (as in mount) share it. It is released when the
stream reads the chunk to its end, moves to another chunk (including one
served from the chunk cache), or falls back to random reads. A buffer is
dropped once no stream pins it and no read is in progress, if it was
consumed or its last stream left it; a read still in flight when the
stream leaves drops it on detach, as UnCache did via destroy. Eviction by
slot limit and memory budget is unchanged.
lastChunkFid is now guarded as well: concurrent ReadAt calls raced on it.
Tests: two ChunkReadAt instances streaming one object in interleaved
slices fetch each chunk exactly once (2-3 times before); leaving a chunk
for a chunk-cache hit or while another read is in flight releases it;
concurrent ReadAt calls on one ChunkReadAt leave no pins behind under
-race.
2026-09-28 21:56:04 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: remove staged EC generation files on teardown and shard delete
The 2PC generation switch stages each run as <base>.ecNN.v<N> plus
versioned .ecx/.ecj/.vif files. Nothing on the volume server removes
them: isEcDataShardFile only recognises the exact .ecNN name, so the
staged files are invisible to every bookkeeping pass, and even
full_teardown's wipe-all path left them behind. Each re-encode therefore
leaks a full shard set per shard-holding disk.
RemoveEcGenerationFiles sweeps <base>.ec*.v<N> and <base>.vif.v<N>,
optionally keeping generations at or above a threshold; teardown and the
reconcile wipe remove every generation, and a per-shard delete removes
that shard's staged generations too.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: delete staged EC generations older than N via VolumeEcShardsDelete
After a 2PC generation switch commits, the superseded generation's
<base>.*.v<N> files sit on disk with no cleanup path: teardown removes
everything, and a per-shard delete only touches the named shards, so the
executor had no RPC that reclaims just the staged leftovers.
delete_generations_older_than removes staged generation files strictly
below the threshold on every disk. Versioned files are never mounted, so
nothing is unloaded first; the committed generation and the canonical
files are preserved.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: mirror staged EC generation cleanup
Parity with the Go volume server: remove_ec_generation_files sweeps
<base>.ec*.v<N> and <base>.vif.v<N> staged by the 2PC switch, called by
remove_ec_volume_files (which covers both teardown paths) and the new
delete_generations_older_than request field; delete_ec_shards removes a
shard's staged generations along with the canonical file.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: match staged generation filenames literally
filepath.Glob interprets metacharacters in the collection part of the
base name, so a collection like a[bc] could match another volume's
staged files (or miss its own). Scan the directory and compare names
literally instead, mirroring the Rust read_dir implementation.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: report generation-sweep errors and drop the store lock first
- snapshot the location base names under the read lock and run the
filesystem sweep after dropping it, so a slow disk cannot stall the
store;
- record per-entry read_dir errors in remove_ec_generation_files and
propagate them from remove_ec_shard_generations instead of flatten()
skipping them;
- warn when a staged-shard generation fails to delete rather than
reporting success with files left behind.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: fail shard delete when the staged-generation listing fails
A transient ReadDir failure fell back to removing canonical shard names
only: staged .v<N> files survived while the RPC still reported success,
leaving the leak invisible to retrying callers. ENOENT still means the
disk simply has no such directory; other listing errors now propagate.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: propagate staged-generation removal failures
delete_ec_shards logged remove_ec_shard_generations errors and the RPC
returned success while staged .v<N> files remained, diverging from the
Go handler which surfaces the failure. The sweep keeps processing the
remaining shards, retains the first error, and volume_ec_shards_delete
maps it to Status::internal so callers can retry.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: notify state change even when the shard sweep errors
delete_ec_shards already deletes and unmounts the shards before
returning a staged-generation failure, so returning early skipped
volume_state_notify and the master kept routing to them until the next
heartbeat. Notify before propagating the error.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: make volume capacity reservation timeout configurable (#11426)
* master: expire reservations on reads, fix int timeout units
- AvailableSpaceForReservation now expires reservations too: a node that
is full of reservations is filtered out before TryReserveCapacity can
clean them, which stranded expired capacity indefinitely.
- Drop TryReserveCapacityWithTimeout: a per-call timeout lets one caller
expire another's live reservations, and the Node interface stays
stable for implementations outside this tree.
- parseReservationTimeout no longer routes integer values through
GetDuration, which read them as nanoseconds; bare numbers are
seconds. The 5m fallback is now the shared DefaultReservationTimeout.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3: do not promote ?prefix into the object for non-List actions
authRequestWithAuthType mapped an empty object to the prefix parameter for
every action, so PUT /bucket?versioning&prefix=x authorized as Write:bucket/x.
An object-scoped grant (Write:bucket/*) could then change bucket versioning,
lifecycle, cors, and object-lock configuration, and the promoted object also
made ResolveS3Action report s3:PutObject to attached IAM policies.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: treat GET ?uploads as a bucket listing for authorization
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: resolve the listing action through the bucket-level object
resolveS3AuthTarget fed the promoted prefix to ResolveS3Action, so a
bucket-level ?uploads request resolved as s3:GetObject on the prefix ARN
in the admin explicit-deny check. Resolve both action and resource
against the object the bucket listing actually scopes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: resolve the listing action through the bucket-level object in AuthorizeAction
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: drop the unreachable object-level uploads case from the resolver test
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-28 07:17:58 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* s3: a list marker that sorts past the prefix leaves nothing to list
AWS scopes a listing to keys under Prefix; StartAfter, Marker and
continuation tokens only reposition inside that range. A marker that
diverges from the prefix at a larger byte is after every key the prefix
can match, so the page is empty. normalizePrefixMarker used to keep such
a marker as the walk cutoff at the bucket root, where the walk descends
into the marker's own directory and returns keys the prefix never names.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep the listing variant's action when a prefix is promoted to object
authRequestWithAuthType promotes ?prefix= into the object argument for
the legacy CanDo path. ResolveS3Action treats a non-empty object as
object-level, so a bucket-level ?versions or ?uploads request carrying a
prefix missed its specific action and fell back to the base List action:
an s3:ListBucket grant then covered s3:ListBucketVersions, and an
explicit Deny on the specific action was skipped on the same path.
Resolve the action against the same bucket-level object the resource
ARN already uses.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: treat GET ?uploads as a bucket listing for authorization
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Update weed/s3api/auth_credentials.go
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* s3: ignore empty intermediate directories in bucketHasUserObjects (#11490)
* s3: keep nested reserved-named dirs from hiding user objects
Reserved folders (.uploads, *.versions) are internal only at the bucket
root; deeper entries with those names are user key prefixes and must be
walked. Also treat a missing subdirectory as empty via isFilerNotFound
(list errors cross gRPC as status errors, not the sentinel), let names
containing backslashes count as objects, and walk iteratively so empty
chains deeper than the old scan depth no longer report non-empty.
* s3: treat reserved-named directories as internal at every level
Object listing interprets .uploads and *.versions directories as
internal storage wherever they appear, so walking them during the
emptiness check would report invisible version remnants as user objects
and block deletion. A reserved name on a file still counts, matching
listing which only special-cases directories.
* s3: count explicit directory objects under reserved names
A directory object created by PutObject (MIME or prefix-object marker
set) is user data even when named .uploads or *.versions; only a plain
directory with a reserved name is internal storage.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-28 06:56:46 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: require dedicated object-lock permissions for x-amz-object-lock-* headers
PutObject, CreateMultipartUpload, and PostPolicy honor the retention and
legal-hold headers after only the route's s3:PutObject check, so a
write-only principal could pin a version under COMPLIANCE retention that
nobody can remove before its retain-until date. On AWS these headers
require s3:PutObjectRetention / s3:PutObjectLegalHold. validateObjectLockHeaders
is the shared funnel for all four call sites; it now authorizes the
corresponding dedicated action when each header is present.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: record the verified POST-policy signer as the request identity
The handler authenticated the form policy signature but stored only the
signer's name, so downstream authorization (the object-lock header check)
re-authenticated the form-signed request as anonymous and evaluated the
wrong principal.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.
The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
volume_copy was one ~400-line handler, and the rule that an existing
local replica is deleted only after the source's ReadVolumeFileStatus
succeeded was held by statement order alone. The keep_remote_data=true
that the pre-copy delete and the failed-copy rollback must share was
kept in sync by a comment pointing from one to the other.
The handler is now a ~60-line orchestrator over connect_to_copy_source,
SourceVolumeStatus::fetch, delete_existing_replica,
plan_copy_destination and a VolumeCopyJob whose run() drives
preallocate_dat, transfer_files, finish_copied_files and
mount_and_reply, with cleanup_failed_copy on error.
delete_existing_replica takes a &SourceVolumeStatus, which only fetch
can construct (private field in a child module), so the delete cannot
be called before the status RPC. Both deletes go through
delete_replica_keep_remote.
Pure refactor: call order, status codes and messages, cancellation
checks, throttling, progress reports and cleanup are unchanged.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix(volume-rust): reserve a disk before replacing a replica in VolumeCopy, and check record counts
Port of the Go VolumeCopy hardening in #11238 and #11252.
- Pick the destination disk before deleting the existing replica, counting
the slot that replica holds as free. If no disk qualifies, the healthy
replica is kept instead of being deleted.
- Read the source's VolumeStatus before and after the copy. When both
succeed and the counts did not change, the mounted replica's file and
deleted counts must match; on mismatch it is unmounted and its files
removed. A failed "before" read skips the check; a failed "after" read
fails the copy.
* fix(volume-rust): let a departing caller cancel VolumeCopy's post-copy status read
Go reads the source's status after the copy with stream.Context(), so the
call ends when the caller leaves. The Rust call had no such link: a
source that stalled there held the copied, unmounted files after the
caller was gone. Race it against the response channel, like the other
blocking steps, so the usual error cleanup removes the partial copy.
remote_storage/s3_tier.rs returned Result<_, String> from every
transfer (upload_file, download_file, read_range[_blocking],
delete_file[_blocking]) and from the tier runtime helpers. The tier
move handlers could only wrap that in Status::internal, so a .dat whose
remote object is gone was indistinguishable from an I/O failure to
weed shell.
Add TierError { NotFound, Io, RuntimeUnavailable, Aborted }. Each
variant carries the existing message verbatim. NotFound follows the
rules remote_storage/s3.rs already uses: raw 404 status on HEAD,
NoSuchKey code on GET; a bare 404 on GET stays Io. A progress-callback
Err becomes Aborted. VolumeError gains a transparent Tier variant and
From<VolumeError> for Status maps Tier(NotFound) to NotFound; the tier
move handlers go through status_with_context, so their message text is
unchanged. Every other tier failure is still Internal.
The remote needle read path keeps io::Error::other, so its error kind
and vacuum's handling of it do not change.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: run the vacuum compaction copy without the store lock
VacuumVolumeCompact held the store write lock for the whole live-needle
copy, including every progress blocking_send on the 16-deep stream. On a
large volume that is minutes with every read, write and heartbeat on the
node parked behind it, long enough for the master to unregister the node.
Split compaction the way Go's CompactByIndex runs it. A short locked step
claims the volume's compacting flag, records the makeup_diff watermark
(index size and compaction revision) and opens fresh .dat/.idx handles.
The copy then replays .idx up to the watermark and copies from those
handles with the store lock released; writes that land meanwhile are
replayed by makeup_diff at commit, as before. The flag is an Arc<AtomicBool>
released when the job is dropped, so every exit path clears it.
Because the flag is now visible to other callers, the operations that would
pull the files out from under the copy refuse while it is set: unmount
(and VolumeConfigure, which unmounts and remounts), delete (checked before
the volume is removed from the map, which a refused destroy used to leave
unmounted), cleanup, and index relocation. A second compact and a commit
stay no-ops, as in Go. The pre-copy fsync is dropped: the copy reads its
own handles through the page cache and .cpd/.cpx are fsynced before commit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume: keep a read-only in-memory index's size for the compaction copy
The unlocked copy replays .idx up to index_file_size(). A read-only
volume whose .sdx could not be built loads its index into memory without
a writer, so that size stayed 0: the copy came out empty and the commit
replaced the volume with it. CompactNeedleMap::load_from_idx now records
the rows it loaded, which is also what Go's IndexFileSize reports for a
read-only index.
The copy's index replay now stops reading at the recorded size instead
of walking rows appended since, which makeup_diff replays anyway.
Adds tests for compacting a read-only volume on both the sorted index
and the in-memory fallback, and for VolumeConfigure stopping when the
unmount is refused during a copy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stop a vacuum copy as soon as its client is gone
The progress callback only noticed a closed response stream when a report
was due, every 128 MiB. With the copy now running outside the store lock,
a copy nobody waits for keeps the volume marked compacting and so keeps
refusing unmount, delete and cleanup until that next report. Check the
stream on every callback.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stream ReadAllNeedles without holding the store lock
read_all_needles held store.read() while Volume::read_all_needles read
every live needle of the volume into a Vec, and kept holding it through
the whole blocking_send loop. Memory grew with the volume, and a slow
client parked the scan in a send with the guard held; needle writes and
the heartbeat take store.write() on a writer-preferring lock, so the node
stopped serving until the client caught up.
Take a DatScanPlan (fresh .dat open, end bound) under a short guard and
walk it with the guard released, sending one needle at a time. Each
record is checked against the live needle map under a brief read guard,
as the scan reaches it, and only a live record is parsed, so a damaged
stale copy does not fail the stream. Records appended while a pass ran
are walked by a follow-up plan, so a needle overwritten during the scan
is streamed once, as its new copy. A vacuum commit or re-create of the
volume during the scan fails the stream, since the map's offsets no
longer describe the pinned file; the plan carries the volume instance
and compaction revision for that check.
DatScanPlan::scan_records yields records unparsed; scan keeps its
behaviour on top of it. Volume::read_all_needles has no caller left and
is removed; its tests move to the RPC.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: check ReadAllNeedles liveness only once the send can proceed
The per-record liveness check ran before blocking_send, so a scan parked
on a full channel held a record it had already judged live. An overwrite
landing during that park left the old copy in the stream, and the
continuation over appended records then streamed the new copy as well.
Reserve channel space first, then take the store read guard, check the
record against the needle map and enqueue it through the permit before
releasing the guard. The wait for space still happens without the lock;
the record is parsed before the guard is taken, and its parse error only
counts if the record turns out to be live.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: collect heartbeats under the store read lock, off the runtime
Every pulse tick, options change and volume-state notification took
store.write() for a whole heartbeat pass, directly on the async heartbeat
task. The pass fstats every volume's .dat twice and hashes its report, so
on a server with many volumes it held the store exclusively for the whole
scan: reads and writes stalled, and with the writer-preferring RwLock a
pending pass parked every new reader too.
The pass only needs to mutate the store for a few rare actions: removing
expired EC volumes, deleting expired volumes past their removal delay, and
setting no-write on IO-quarantined volumes. It now runs under store.read(),
records those as (disk, volume id) actions, and applies them afterwards
under a short store.write() that is only taken when there is something to
do. Each action re-checks its target under the write lock, so a volume
written to, replaced or removed in between is left alone. Expired EC
volumes are still removed before the volume pass, as before, because the
EC shard count feeds the disk-space-low max volume count. Every pass runs
on the blocking pool via spawn_blocking.
The heartbeat message is unchanged for the same store state.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: take has_no_ec_shards with the heartbeat's volume list
The heartbeat pass took has_no_ec_shards from the EC phase's read lock,
then built the volume list under a second one. An EC shard mounted in
between went out as "no EC shards" beside a volume list taken after the
mount, and the master clears a server's EC registrations on that flag.
has_no_ec_shards is now computed under the same read lock as the volume
list, with the EC phase's filter: not expired, not quarantined, at least
one shard.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: let the sweep release volumes that stay empty and quiet
Vacuuming reclaims bytes but not slots: a fully emptied volume stays
registered to its collection forever, and since growth is gated only on
slot count a store at 99% free disk can still refuse writes to other
collections (#11429). volume.deleteEmpty exists but is manual-only.
With -vacuumDeleteEmptyAfterSeconds (or master.vacuumDeleteEmptyAfterSeconds
under weed server/mini; default 0, off) the automatic sweep now deletes
replica copies that have stayed empty and quiet for that long, the same
rule volume.deleteEmpty applies on demand: remote-backed copies are
skipped, and every delete carries the volume server's onlyEmpty /
onlyGarbage guards so a copy written since the last report is refused
rather than removed. Copies that still hold data or were written
recently stay; only a volume whose every copy is deleted leaves the
sweep's work map, sparing a compaction of bytes that are all deleted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: harden empty-volume sweep against partial and racing deletes
Review follow-up on #11477:
- delete a volume only when every replica copy is a verifiable
empty-and-quiet candidate; deleting the empty copy of a volume whose
sibling holds live files would silently cut its replica count
(greptile P1).
- drain the volume out of the writable list before deleting, the same
drain the compact pass uses, so PickForWrite stops assigning it and
pending writes settle (devin).
- bound the VolumeDelete RPC so one stalled server cannot hold the
vacuum lock indefinitely (greptile P1, reusing allocateVolumeTimeout).
The vid2location panic scenario raised in review does not exist:
VolumeLocationList methods are nil-receiver safe and a missing vid just
fails enoughCopies, so a partially deleted volume skips compaction
instead of crashing the sweep.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: unregister deleted empty replicas and prune the sweep list
A successful VolumeDelete only updates the volume server; the master
still tracked the replica and kept it in the sweep's location list for
the compaction pass (coderabbit on #11477). Unregister the replica right
after its delete succeeds and drop it from the sweep copy, so a partially
deleted volume only compacts copies that still exist.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: pin deleting volumes out of the writable list across heartbeats
Review follow-up on #11477 (greptile): DrainAndRemoveFromWritable only
removed the volume once; a heartbeat landing between the drain and the
replica deletes re-evaluated writability and re-added it, so a client
write could reach a replica whose siblings were already gone and leave
the volume under-replicated when the last copy refused its onlyEmpty
delete.
MarkDeleting records the vid in deletingVolumes — checked inside
setVolumeWritable so heartbeat, capacity-recovery, and admin re-add
paths all hold it out — and UnmarkDeleting releases it once the sweep
finishes the copy pass. A partially deleted volume's surviving replicas
then return to writable through the normal heartbeat path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: restore writability when a sweep delete survives
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-27 07:05:22 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: record requester identity before the authz verdict for audit
Identity was only stored in request context on the success branch, so
denied requests reached WriteErrorResponse without requester attribution
and audit entries had empty requester/requester_arn/requester_identity.
Authentication failures still resolve no identity, so unauthenticated
denials stay unattributed.
Fixes#11474
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep the resolved identity through authz denial in Auth
Review follow-up on #11479 (devin): authRequest discarded the identity
on every error, so a request that authenticated fine but failed the
action check still reached handleAuthResult with no identity and the
deny path could not audit a requester. Auth now calls
authRequestWithAuthType directly, the same entry AuthPostPolicy uses,
so the resolved identity reaches the error writer; a failed authN
still resolves no identity and stays unattributed. The regression test
now signs a denied request end to end through iam.Auth.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-27 07:03:01 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sts: export CalculateSessionDuration
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: honor configured session bounds on AssumeRole and LDAP identity
prepareSTSCredentials hardcoded a one-hour session when the caller
omitted DurationSeconds, so sts.tokenDuration was ignored and
sts.maxSessionLength only clamped explicit requests: asking for 3600s
against a 20m ceiling was rejected while omitting the parameter was
granted a full hour (#11473). The two affected handlers now use the
same default-then-cap calculation as AssumeRoleWithWebIdentity.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: keep MaxSessionDuration through role store copies
copyRoleDefinition rebuilt RoleDefinition field by field and dropped
MaxSessionDuration, so memory-backed role stores silently discarded the
per-role session bound on every write and read (devin on #11478).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sts: apply per-role MaxSessionDuration to resolved session durations
Review follow-up on #11478 (devin): the role bound only ever applied to
explicit DurationSeconds values — an omitted duration resolved to the
configured default and sailed past a shorter role max on every assume
path.
- capDurationByRole now resolves min(requested||tokenDuration, roleMax),
so AssumeRoleWithWebIdentity and AssumeRoleWithCredentials cap
defaults the same way they cap explicit values.
- prepareSTSCredentials caps the calculated duration at the named
role's MaxSessionDuration, covering the AssumeRole and LDAP handlers;
self-assumption has no role definition to consult.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: keep MaxSessionDuration through the cached role store
genericCopyRoleDefinition drops MaxSessionDuration the same way
copyRoleDefinition did, so the cached filer role store reads back a zero
maximum and every downstream duration cap is skipped (greptile on
#11478).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sts: only materialize defaults that pass session duration validation
Review follow-up on #11478 (greptile): materializing an omitted
DurationSeconds into an explicit value could exceed the service's own
input bound (a configured tokenDuration above maxSessionLength) and turn
a previously working request into a validation error.
capDurationByRole now leaves nil anything the service can resolve
better itself, clamps a tightened default at maxSessionLengthSeconds,
and floors a role bound below 900s to the tightest issuable value.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-27 07:01:51 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep dedicated object-lock actions pinned during action resolution
A coarse action that already names a dedicated operation (governance
bypass, retention, legal hold, bucket object-lock config) now resolves to
itself before request shape is consulted. Previously a synthetic
DELETE ?versionId authorization request re-resolved to
s3:DeleteObjectVersion, so the bypass check was satisfied by the
delete-version grant alone; with the pin it evaluates
s3:BypassGovernanceRetention as intended.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: cover pinned object-lock actions against competing query params
Locks in the resolution for every dedicated action in the pin set, incl.
the retention and legal-hold shapes carrying versionId.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: stamp entries with IF_CHUNKS_EQUAL so a stale write-back cannot delete live chunks
updateLocalEntry records the RemoteEntry stamp after an upload by writing the
event's entry back with UpdateEntry. The filer deletes every stored chunk
absent from an updated entry, so when the file was rewritten while its upload
was in flight (or the event is a replay), the stale snapshot deletes the
rewrite's chunks: the entry then points at the new fid with no needle behind
it, and the rewrite's own upload fails and is skipped as superseded.
The stamp write now carries WriteCondition IF_CHUNKS_EQUAL over the event's
chunk fids, evaluated by the filer under the path lock. A refused stamp means
the filer moved past this event; the superseding event follows in the log and
stamps the current entry, so the refusal is logged and skipped like a
superseded upload.
Reproduction: weed server -filer plus a weed server -s3 remote, remote.mount,
filer.remote.sync; hold the remote (docker pause) so one upload stays in
flight, rewrite the file through the filer, unpause. Before: the entry's chunk
is 404 on every volume server. After: the stale stamp is refused, the rewrite's
chunk stays live and reads back after a vacuum.
* filer.remote.sync: stamp entries with IF_ENTRY_EQUAL so stale inline content or metadata cannot be restored
The IF_CHUNKS_EQUAL guard compared only the chunk fid multiset, so a
rewrite that touched inline content or metadata alone still compared
equal and the stale snapshot overwrote the live entry. The new clause
compares the whole stored entry against the event's entry under the
same path lock.
* filer: route conditional UpdateEntry to the entry's owner filer
Two filers locking the same path locally could still pass a stale
condition on the non-owner while the owner's entry had moved on. When a
condition or expected_extended precondition is set, forward the request
to the entry's owner the same way conditional CreateEntry does, with
is_moved bounding the hop.
* filer: compare IF_ENTRY_EQUAL against the normalized expected entry
FindEntry grows FileSize to the chunk extent, so a raw event entry with
FileSize still zero failed the condition on an unchanged file and the
stamp was skipped, letting a replay upload the object again.
* filer.remote.sync: classify refused stamps by gRPC status only
A FailedPrecondition substring in an unrelated error would have been
swallowed as a skipped stamp; status.FromError already unwraps.
* remote sync: keep the event entry intact for IF_ENTRY_EQUAL
* filer: honor is_moved only from ring member connections
is_moved is caller-controlled, so a request could set it to skip owner
routing and run a conditional check under a non-owner's lock. Verify the
marker against the peer's connection address and the lock ring members;
an unverified marker is ignored and the request routes like a fresh one.
* filer: refuse unverifiable is_moved at a non-owner, cache ring IPs
Follow-up fixes from review on the is_moved provenance check:
- checkMovedMarker replaces "ignore and re-forward" for markers that did
not arrive on a ring member's connection. Re-forwarding a claimed hop
could cycle while rings disagree; instead the request is refused with
FailedPrecondition unless this filer is the key's owner, in which case
applying locally is correct anyway.
- ringMemberIPs caches resolved member addresses per ring membership so
hostname-advertising deployments do not pay a DNS lookup per forwarded
request; failed lookups are not cached so a DNS blip self-heals.
- DistributedUnlock no longer dereferences the nil response of a failed
next-hop RPC.
* filer: refuse unverifiable is_moved with PermissionDenied, not FailedPrecondition
A routing refusal is different in kind from a write-condition mismatch:
remote sync treats FailedPrecondition as a stale stamp and skips it, so
reusing that code let a routing failure pass as synced. Owner checks now
also run before the peer-IP lookup so the common accept path does no DNS.
* filer: expire resolved ring member IPs after 5 minutes
A member's hostname can re-resolve to a new IP while its ring address
stays unchanged; caching forever would reject its genuine forwards until
a membership change or restart.
* filer: deduplicate concurrent ring member DNS lookups
At cache expiry, parallel forwarded requests would each resolve every
member hostname serially; singleflight collapses them into one lookup
per ring membership.
* filer: detach the shared ring lookup from the caller's context
The singleflight winner's ctx is cancelled when its request ends; the
shared result would then be an incomplete member list and genuine
forwards denied. The lookup now runs on a detached context with its
own deadline so a canceled caller cannot poison it.
* filer: resolve ring member hostnames in parallel
The shared lookup gave every member one serial budget, so a few slow
resolutions could leave later members out of the cached list and reject
their genuine forwards. Each member now resolves concurrently under its
own detached deadline.
* filer: gather literal member IPs before spawning lookups
A ring mixing IP literals and hostnames raced: the literal appends ran
unlocked alongside the resolver goroutines' locked appends. Split into
two passes so only hostname results share the mutex.
---------
Co-authored-by: jsas <1351492+jsas@users.noreply.github.com>
2026-09-26 19:42:23 +08:00
Mohd Quamar TyagiGitHubcoderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>Chris Lucoderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* volume: skip TTL last-write scan when it cannot fit its budget
* Update weed/storage/volume_checking.go
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* s3: evaluate bucket policy before ACL public-read for anonymous requests
AuthWithPublicRead granted anonymous access on a public-read ACL before
consulting the bucket policy, so an explicit Deny (e.g. s3:ListBucket)
was skipped for anonymous callers while still enforced for authenticated
ones. Run the policy engine first: a matching Deny or Allow is honored,
otherwise fall through to the ACL grant as before.
* s3: defer object-level anonymous requests to the handler's policy recheck
Evaluating the bucket policy with a nil entry at middleware time makes
tag conditions like s3:ExistingObjectTag/<key> resolve against missing
values, so a conditional Deny could wrongly block anonymous Get/Head on
a public bucket whose handler recheck would permit it. Object requests
now take the ACL grant and let Get/HeadObjectHandler re-evaluate with
the fetched entry; only bucket-level requests (List, HeadBucket), which
have no such recheck, are decided by the middleware policy verdict.
Reading the bucket config first also refreshes the compiled policy on a
cache miss, so a remotely deleted policy cannot leave a stale verdict
in the engine for nonresident buckets.
* s3: recheck bucket policy before serving directory objects
handleDirectoryObjectRequest runs before the object handlers' policy
recheck, so directory content on a public-read bucket was served to
anonymous callers without any policy evaluation. Evaluate the policy
with the directory entry, matching the recheck the file path performs.
2026-09-26 17:59:57 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: hold back last chunk until CRC verifies on whole-needle reads
Above PagedReadLimit the needle is streamed: headers and body go out
before the checksum is computed, so a corrupted needle was served as
200 with bad bytes and readers could not fall back to a replica. The
final chunk is now written only after the checksum verifies; on a
mismatch the response ends short of Content-Length and the client sees
a failed transfer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: verify needle checksum on streamed reads (parity)
Mirror the Go fix: carry the needle checksum in NeedleStreamInfo and
have StreamingBody accumulate the CRC and verify it before emitting the
last frame; a mismatch ends the body with an error so the client sees
the transfer fail rather than receiving corrupt bytes that look
complete.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* server: abort the transfer when a committed stream fails
A writeFn failure after any byte or WriteHeader call leaves the declared
status and Content-Length already sent; http.Error's text then joins the
body and can exactly fill the withheld tail of a corrupted needle read —
the client sees a complete 200 instead of a failed transfer to retry.
Track whether the response is committed (headers sent, or bytes buffered
for the deferred flush) and panic with http.ErrAbortHandler instead of
appending an error body; pre-commit failures keep the 500 path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* server: drop the response writer wrapper from the committed-response check
Counting buffered writes is enough: with no bytes buffered the status and
headers cannot have gone out, and the range branches commit via the
explicit WriteHeader call before writeFn runs. The extra ResponseWriter
wrapper added a new Write sink site that CodeQL flags.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 16:50:41 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: scrub live needles' stored id against the index key
scrubVolumeData only compared the needle's stored id for tombstones, so
header damage on a live needle — where the data CRC cannot see it —
passed every scrub mode while reads of that needle kept failing or
serving the wrong key's data. Compare the id for every indexed needle.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: scrub live needles' stored id against the index key (parity)
Mirror the Go scrub fix: compare the stored needle id with the index
key for live needles too, not only for deleted ones.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: cover damaged live needle id in scrub test
The tombstone test proved the index-key check fires for deleted entries;
add the live-needle mirror of Go's TestScrubVolumeDataChecksLiveNeedleId
so a regression in the live path is caught in Rust too.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 16:10:14 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: serialize every ViperProxy method; stop promoting unlocked viper calls
ViperProxy embedded *viper.Viper, so only the five declared methods took
the mutex while every promoted call — GetStringMap in backend.LoadConfiguration
was the reported crash — touched viper's maps unsynchronized. `weed server`
starts the volume server (SetDefault writer) and the master (GetStringMap
reader) back to back, and a race build reports the pair on a plain start.
The wrapped viper is now a named field: a method must be declared here to
exist on the proxy, so unsynchronized access fails at compile time rather
than at runtime. Every promoted use in the tree (GetStringMap, GetUint32,
GetFloat64, GetDuration, IsSet, AllKeys, Set) gets a locked wrapper;
NewViperProxy replaces struct literals for local vipers. GetStringMap
deep-copies its result — viper hands back the internal subtree, so
iterating it after the lock is released would race the next writer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: take the shared lock while LoadConfiguration merges a config file
viper.MergeInConfig rewrites the same maps the proxy serializes; without
the lock a merge can race a concurrent SetDefault or reader exactly like
the reported startup crash.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: deep-copy slice elements in the GetStringMap snapshot
A slice of maps inside the returned subtree still shared the inner maps —
copy elements recursively so nothing the caller mutates is viper's
internal state.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: add the missing AutomaticEnv wrapper used by tests
sse_reader_test reaches it through GetViper(); without the wrapper the
call no longer exists once the viper field stopped being embedded.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: return a fresh slice from GetStringSlice
A stored []string comes back uncast from viper — the backing array is
shared internal state like the GetStringMap subtree, so copy it while
holding the lock.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: do not finish a GET when the needle CRC mismatches
A streamed full-needle read compared the CRC only after every page had been written. Once the response buffer flushed, the client already had a completed 200 and the corrupt bytes. Hold the last page until the checksum matches, and if an earlier page has already been flushed, abort the connection instead of calling http.Error.
Fixes#11459
* volume server: abort partial-content bodies on write error too
The non-Range path drops the unflushed tail and aborts on a mid-body
error; the single-range and multi-range paths still flushed it after
WriteHeader(206) was committed, delivering corrupt bytes as a complete
body.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: assert the started 200 is aborted in the write-error test
The test previously returned on any request error, so it passed without
verifying the abort. It now asserts the client got the committed 200
headers and then a failed body read. Also trims comments.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 16:00:22 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: let volume.deleteEmpty drop volumes with no live needles
The candidate check only accepted a .dat at superblock size, so a volume
whose every needle was deleted still had to be vacuumed first — minutes
of compaction to rewrite bytes that were all garbage anyway. FileCount
counts every indexed entry and DeleteCount every entry made garbage by
overwrite or delete, so FileCount <= DeleteCount means nothing live
remains and the volume can be unlinked directly. The quietFor guard is
unchanged.
* volume server: add only_garbage VolumeDelete guard
VolumeDelete(only_empty) refuses every volume that ever held data, so a
volume whose needles are all deleted could only be removed after a
vacuum rewrote it. The new only_garbage flag deletes only when the byte
counters show nothing live: DeletedSize covering all of ContentSize, the
same all-garbage state vacuum measures. Byte counters are used because
the file/delete counts drift on index reload.
* rust volume: mirror only_garbage VolumeDelete guard
Same check as the Go server: a volume deletes under only_garbage when
its deleted bytes cover all content bytes. The grpc handler rejects
before the store drops the volume from its map, since destroy errors
after removal would still unmount it.
* volume delete: let either enabled check pass, keep onlyEmpty on the wire
An upgraded shell sending only_garbage to a pre-upgrade server would be
read as an unconditional delete (field ignored, only_empty false). The
request now keeps only_empty set so old servers check emptiness and
refuse, while new servers delete when either check passes.
* volume.deleteEmpty: skip remote-backed and protected read-only volumes
A remote-tiered replica shares its cloud object with the other replicas,
so keepRemoteData=false on one delete removes data they still reference.
Protected read-only volumes are quarantined or under maintenance, which
is exactly when a replica should not be dropped.
* volume delete: validate guarded copies across disks before deleting
* volume delete: hold copy locks across guarded validate-and-delete
CheckVolumeDeletable released each copy's locks before Destroy ran, so a
write landing on a later copy between the two passes refused its destroy
after earlier copies were already removed. Pin every copy's
dataFileAccessLock (and its location's volumesLock) across validation and
removal so a refused delete leaves all copies intact.
* volume delete: send deleted-volume notices after releasing locks
A blocking send on a full DeletedVolumesChan under volumesLock can stall
the heartbeat loop that drains it while it waits on the same locks.
Collect the notices under the lock span and send after release.
* pb: restore generated-file cosmetics to match the repo's protoc version
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: one long-lived tokio runtime for blocking tiered S3 reads
block_on_tier_future, behind read_range_blocking and delete_file_blocking,
spawned an OS thread and built a fresh current-thread tokio runtime on every
call, then tore the runtime down. On an S3-tiered volume that is once per
needle read, per streamed 64 KiB chunk, per DatScanPlan record and per
destroy. The SDK client's pooled HTTPS connections are driven by tasks on the
runtime a request ran on, so each teardown dropped the pool and every call
re-dialed and re-handshook TLS. A panic inside the SDK was also flattened to
the fixed string "tier runtime thread panicked".
Now one process-wide runtime (OnceLock, multi_thread, 2 workers named
tier-io) drives all tier I/O; block_on_tier_future spawns onto it and parks
the caller on an mpsc channel for the JoinHandle result. Blocking the caller
is unavoidable (the storage layer is synchronous) and is what the old code
did through thread::spawn().join(). Handle::block_on is not used because the
wrappers are also reached from inside another runtime's worker, where it
panics with "Cannot start a runtime from within a runtime". JoinError panics
are downcast to &str/String and the payload is kept in the error.
Tests cover runtime reuse (Handle::id equal across calls, thread name
tier-io), calls from a std thread, from spawn_blocking, and directly from
current-thread and multi-thread runtime contexts, and the panic payload.
Against the old body 7 of 9 fail.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: return a tier runtime build failure instead of panicking
Review follow-up. tier_runtime() expect'ed the runtime build, so an OS
refusing threads panicked inside Volume::destroy (after the volume left
the in-memory map, before its files were removed) and inside needle
reads, bypassing their error paths. Keep the runtime in a
Mutex<Option<Runtime>> behind tier_handle() -> Result<Handle, String>:
a failed build is returned to the caller through block_on_tier_future's
existing Result and is not cached, so a later call retries once the
pressure is gone. The lock is held only while building.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: trim comments on the shared tier I/O runtime
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: VolumeEcShardsDelete full teardown unloads every disk and keeps the shard gauge honest
Go's VolumeEcShardsDelete full teardown calls vs.store.UnloadEcVolume in
the blanket path (weed/server/volume_grpc_erasure_coding.go:488) and
location.UnloadEcVolume in the generation-fenced path (:511): each disk
that had the volume registered drops it, closes its shard descriptors
and gives back its ec_shards gauge before the artifacts are unlinked.
The Rust handler used Store::remove_ec_volume / DiskLocation::remove_ec_volume
instead, which only remove the map entry. Store::remove_ec_volume also
stops at the FIRST disk holding the vid, so on a split-disk volume
(shards on several disks) the blanket teardown left the sibling disks'
EcVolume registered with open fds while the unlink loop deleted their
files underneath it: the heartbeat kept advertising shards whose files
were gone, the inodes stayed pinned by the open descriptors, and the
VOLUME_GAUGE{collection,"ec_shards"} never came back down. The fenced
path leaked the gauge and the descriptors the same way on the one disk
it wiped.
Both paths now use the unload_ec_volume helpers from #11413 (every disk
for the blanket teardown, the strictly-older disk for the fenced one),
and the two Status::internal messages name the disk directory like Go's
"... on %s: %w".
Regression tests build a two-disk store with the same vid mounted on
each disk (the SplitDiskEcFixture, which gains a collection knob so the
gauge read is isolated from parallel tests mounting under "") and assert
that a blanket teardown leaves no EcVolume registered on any disk and
returns the gauge to its pre-mount value, and that a fenced teardown
decrements the gauge for the older disk's shard while preserving the
newer disk. Both fail against the previous handler.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume server: trim comments on the EC full-teardown unload path
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: reject invalid -minFreeSpace and duration flags at startup like Go
Symptom: the Rust volume server swallowed invalid values for three
startup flags and ran with a limit the operator did not ask for.
-minFreeSpace=150 became 150 bytes (Go: fatal, percent must be 0..=100),
-minFreeSpace=10GiBx and -minFreeSpace=50B fell through to the 1%
default (Go: fatal), and -inflightUploadDataTimeout=30sec, =abc or =30
silently became the 60 s default (Go: flag.Duration usage error, exit 2).
Go's byte-size units (1,024MB, 42 mib, 100Ki, 1eb) were also not
understood and mapped to 1%.
Go reference: weed/util/minfreespace.go:73-106 (MustParseMinFreeSpace /
ParseMinFreeSpace), weed/util/bytes.go:212-243 (ParseBytes and
bytesSizeTable), weed/command/volume.go:226-227 (flag.Duration, i.e.
time.ParseDuration grammar).
Fix: the two inflight timeout Cli fields are now std::time::Duration
with a clap value_parser that ports time.ParseDuration (units ns/us/µs/
μs/ms/s/m/h, fractions, bare 0, int64-nanosecond overflow), so clap
prints a usage error naming the value and exits 2 at the same layer as
Go's flag package. parse_min_free_spaces returns Result and is built
from Go-parity parse_min_free_space / parse_bytes; resolve_config_with_env
aborts with Go's "The value specified in -minFreeSpace not a valid
value" message, the fail-fast pattern it already uses for --max, --index
and --readMode. No new dependency. Negative durations and NaN percent
are rejected explicitly (Go quirks with no sane Rust representation).
Tests cover the Go fixtures from minfreespace_test.go and bytes_test.go,
the duration grammar, and flag-level rejection through Cli::try_parse_from
and resolve_config_with_env; the flag-level tests were confirmed to fail
against the previous parser.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: say that -minFreeSpace commas separate directories, never thousands
Review follow-up. parse_bytes keeps Go's ParseBytes thousands-comma
support, but the flag is split on commas into per-directory entries
before any entry is parsed (Go's MustParseMinFreeSpace does the same),
so -minFreeSpace=1,024MB is the two entries 1 and 024MB. The doc
comment and the single-entry tests advertised the comma form as if it
were reachable from the flag. Document the split and add a flag-level
test: two entries from parse_min_free_spaces, and resolve_config
aborting on the directory-count mismatch with one -dir.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: trim comments on the strict flag parsing
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 12:00:06 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: restore the HTTP REST API reference
The REST API documentation was lost in the README refactor, and the wiki
only covers the master server API. Add a complete reference for the three
HTTP surfaces — filer paths, master file-id/topology endpoints, and
volume-server content endpoints — generated from the actual handlers and
query parameters.
* docs: correct REST API details flagged in review
Tagging uses Seaweed- headers not query params, the filer recursive
delete option changes the DELETE default, omitted resize mode does not
mean fit, default file mode is 0660, the master redirect is 308, the
listing flag is -dirListLimit, TUS is enabled by default at /.tus, and
-port.public opens the separate read-only listener.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: match the tagging delete example to canonical header names
Go canonicalizes Seaweed-k1 to Seaweed-K1 on write, and the delete list
is compared case-sensitively, so ?tagging=k1,k2 would not match.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: detect S3 not-found by typed SDK errors, not the "service error" string
remote_storage/s3.rs decided ObjectNotFound by matching the Display
output of an aws_sdk_s3 SdkError against "NoSuchKey" / "404" /
"NotFound". In the locked SDK (aws-smithy-runtime-api 1.11.6,
src/client/result.rs:487-497) that Display is a fixed string per
variant, "service error" for every S3 error, so ObjectNotFound was
unreachable: every missing remote object surfaced as
Other("s3 get object: service error") with the real cause discarded.
Go (weed/remote_storage/s3/s3_storage_client.go) uses typed checks:
HEAD (373-374): awserr.RequestFailure with StatusCode() == 404;
GET (436-437): awserr.Error with Code() == s3.ErrCodeNoSuchKey.
read_file now matches SdkError::ServiceError whose GetObjectError
is_no_such_key(); a bare 404 on GET stays a generic error, as in Go.
stat_file matches HeadObjectError::is_not_found() or a raw HTTP 404
status, Go's actual condition. Non-service errors fall through to
Other unchanged. Every SdkError message in s3.rs and s3_tier.rs is
formatted with DisplayErrorContext so the S3 error code and message
survive instead of "service error".
Six network-free unit tests drive the client through a canned
HttpClient (404 NoSuchKey, bare 404 on GET and HEAD, 404 with a
foreign body on HEAD, 403 AccessDenied on GET and HEAD). They need
aws-smithy-runtime-api as a dev-dependency; it is already in the lock
at a single version, so no new crates.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: HEAD not-found is the raw 404 status alone, as in Go
Review follow-up. The HEAD arm also accepted the SDK's NotFound error
code on any status, so a 400 carrying <Code>NotFound</Code> became a
missing object. Go's stat looks only at RequestFailure.StatusCode() ==
404 (weed/remote_storage/s3/s3_storage_client.go:373); do the same. The
raw status still covers the body-less 404 the SDK turns into NotFound
and a 404 whose body names a foreign code. Regression test for the
non-404 NotFound body, which failed against the previous arm.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: trim comments on the typed S3 not-found checks
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: franchb <ru.balloo@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: exclude marker part in listObjectParts pagination
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* s3api: guard listObjectParts marker boundary and enhance pagination test
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* s3api: fold in review feedback from the parallel #11462 fix
Same core fix; this adds the explanatory comment, tightens the overflow
guard to math.MaxInt64, makes the fake filer sort entries like a real
listing, and adds the marker-exclusivity assertions alongside the
pagination walk.
Co-authored-by: yi111 <yi111@users.noreply.github.com>
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: yi111 <yi111@users.noreply.github.com>
2026-09-26 11:58:23 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* cluster: never broadcast an empty lock ring
An empty member list is never a usable ring state, but a delayed
RemoveServer on a former leader can fire after the new leader already
broadcast the recovered ring. That late broadcast carries a newer
wall-clock version, so clients accept the empty ring and permanently
reject the good one.
Skip the broadcast entirely when the member list is empty, keeping the
last non-empty snapshot for reconnecting clients.
* cluster: periodically rebroadcast the lock ring
Ring updates are purely event-driven, so one lost or poisoned update is
permanent until the next membership change — with a single filer that may
never come. Re-arm a per-group timer after every broadcast so the current
leader keeps re-sending the ring; clients reject nothing newer than their
last accepted version, so a re-sent snapshot always heals a stale view.
* filer,s3api: reset the lock ring on master change
Ring versions are per-master monotonic — each master stamps wall-clock
nanoseconds — so a late high-version update accepted from a former leader
makes the new leader's snapshot look stale forever. Detect a leader
change across the reconnect gap (currentMaster is cleared between
attempts, so remember the last served master) and reset the ring to
bootstrap state so the new leader's view always applies.
* cluster: fail lock acquisition when no lock server exists
retryUntilLocked loops forever, so a filer reporting an empty lock ring
wedges every append write indefinitely. Bound only the "no lock server
found" case — ordinary contention is still waited out since the holder
releases eventually. The constructors now return nil on failure: the
filer append path and S3 object writes fail fast, while mounts degrade
to their existing lockless mode.
* cluster: reset only the ring version on master change
Ring versions are per-master monotonic, so a version gate reset is all a
leader change needs. Clearing the whole ring made every filer its own
write owner until the next update and dropped the prior-owner window for
keys the new leader remaps; the last ring now keeps routing until the
new leader's snapshot transitions off it.
* cluster: skip redundant ring installs and defer rebroadcasts
An unchanged member list now only bumps the accepted version instead of
installing a snapshot: periodic rebroadcasts no longer fire the
topology-change callback or restart the prior-owner window. And a
rebroadcast that lands inside a membership stabilization window yields
to the pending timer rather than publishing an intermediate ring.
* cluster,mount: bound lock unavailability, fail ops that cannot lock
Only 'lock already owned' contention retries without bound now; every
other failure — no lock server, or a dead ring member refusing
connections — shares the same unavailability budget, so a ring naming
departed filers can no longer hang a lock forever. Mount open-write,
create, and rename fail with EAGAIN when the required lock cannot be
acquired instead of proceeding without cross-mount serialization.
* cluster: check pending stabilization inside the broadcast critical section
rebroadcast released the mutex between the pending-timer check and
nextBroadcastUpdate, so a membership change arriving in the gap could arm
a stabilization timer while the rebroadcast emitted an intermediate ring.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: acquire path locks before mutating create/rename state
Create took the DLM lock only after the filer create, so a lock failure
returned EAGAIN with an eagerly persisted file left behind. Rename marked
source handles renamed before acquiring locks, so a failed acquisition
left them suppressing old-path flushes for a rename that never happened.
Both now take the locks first; the create's lock is released again if the
entry race loses to another creator and AcquireHandle takes over.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: keep the old-path lock when rename lock migration fails
The migration stopped the handle's lock before acquiring the replacement,
so a nil result left the handle writing with no lock at all. Acquiring the
new-path lock first means failure keeps the existing lock instead of
reporting success with serialization dropped.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: skip new-path rename lock when a handle already holds it
A target file open for write on this mount already carries a lock on
newPath; the lock manager does not grant a second lock to the same
owner, so the rename would wait on itself until the handle closed.
Also avoid locking twice when old and new paths coincide.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: hand the rename's target lock to the migrating handle
The rename holds a lock on newPath for its duration, so the response
migration's fresh acquisition waited on that same lock until the handle
released — under fhLockTable, blocking the handle's own close. Adopt the
rename's lock directly; nested move responses still acquire their own.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: move the replaced target's lock to the renamed handle
When the target path was already locked by an open handle on this
mount, the migrated source handle kept only its stale old-path lock —
the target's close would then release the last lock on the new path
while the renamed handle was still open. Adopt the replaced handle's
lock instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: stop the handle lock inside the fh lock on release
ReleaseHandle stopped fh.dlmLock before taking the fhLockTable slot, so
a rename migration holding that slot could still observe and adopt a
lock that was already stopping. Stopping under the fh lock makes the
transfer serialize against the release.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: claim the replaced target's lock for the renamed handle
When the target path is already locked by an open handle on this mount,
adopting it at migration time keeps the renamed path protected after
that handle closes, without waiting on a lock this mount already holds.
If the handle was released mid-migration the claimed lock is stopped,
and a fresh acquire covers the case where it was already gone.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: claim the target handle's lock before the rename runs
Skipping the new-path lock when a handle already holds it let that
handle's close release the lock mid-rename, leaving the path unguarded
until the response migrated it. Take over the lock at check time and
hold it for the rename's duration: the response adopts it for the
migrating handle, or it returns to the target handle / is released on
failure. The target handle lookup also falls back to the entry's stored
inode for a forgotten path mapping.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: read handle locks only under the fh lock during rename
The loose dlmLock reads raced ReleaseHandle, which now mutates the lock
inside the handle lock; check and claim it under the same hold.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep lazy remote reads from resurrecting deleted paths
Under a remote mount with filer.remote.sync as write-back, a path that
was deleted or renamed away could come back as a chunkless remote-only
entry: between the local delete and the daemon's remote delete, a store
miss made maybeLazyFetchFromRemote trust a bucket that was behind the
filer. The ghost then outlived the remote object -- HEAD answered 200,
GET failed, and nothing cleaned it up.
The filer now tombstones paths it deletes under a remote mount, learned
both synchronously from its own delete path and from peer metadata
events. The lazy fetch and the lazy listing skip a tombstoned path until
the path is written again, until the mount's persisted write-back sync
offset has passed the delete event (the remote delete has landed), or
until a generous TTL covers a mount without a daemon.
Fixes#11440
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: cover recursive remote deletes with an ancestor tombstone
A recursive delete now records the directory tombstone before walking
children, so a partial traversal or a store that drops the subtree
without listing it still leaves every descendant covered. Directory
tombstones also subsume older descendant entries on add, descendant
adds covered by a standing ancestor are skipped, and an existing
tombstone can be refreshed even at capacity.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: scope remote tombstones to the deleted object's generation
A remote object whose own mtime postdates the local delete is a new
generation, not the one the tombstone hides, so a recreated directory
can surface remote writes made after its delete while old-generation
objects stay hidden. Lazy fetch now stats the remote object before
deciding, listings pass each child's remote mtime, and a sync offset
releases a tombstone once it reaches the delete's own timestamp.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: rebuild remote deletion tombstones after restart
In-memory tombstones are lost on restart while remote write-back
offsets persist, so a filer boot replays the persisted metadata log
from the oldest mount offset and folds deletes back into the tombstone
set through the same event handler. Lazy remote reads hold off while
the replay runs so a pending delete cannot resurrect in the gap.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: release remote tombstones only after their delete event lands
The write-back offset orders against event timestamps, but the synchronous
delete path recorded tombstones with the local clock before its event was
emitted — a later unrelated event could already have pushed the mount's
watermark past that guess, releasing the tombstone before the daemon
applied the delete. Tombstones recorded ahead of their event are now
marked pending and can only be lifted by the event confirming them or by
TTL; event-stamped tombstones release through the offset as before.
The remote-mtime generation bypass is dropped: remote and filer clocks
are independent, and a pending remote delete removes whatever object sits
at the path, so a "newer" remote object would only resurrect as a
phantom. Tombstoned lookups now skip the remote stat entirely.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: drop dir tombstone when recursive delete fails before listing
The ancestor tombstone is recorded before the child listing; if that
listing fails nothing was deleted, and the leftover tombstone would hide
still-existing remote children for the whole TTL. Tombstones for children
already deleted stay, since their remote deletes are still owed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: block lazy remote reads on startup tombstone rebuild
The rebuild gate is now a done-channel set synchronously before the
replay goroutine starts, so no lazy read can slip through in between.
Reads wait on it with context cancellation instead of returning an
empty miss that makes remote-only objects look deleted.
The replay start is floored at now-TTL: mounts without a recorded
write-back offset previously replayed the whole persisted history, and
events older than the TTL would only build already-expired tombstones.
The gate check now runs after the mount lookup so replaying the meta
log's own directory listings does not deadlock on the gate, and the
replay retries with backoff until it succeeds instead of failing open.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: mark restamped tombstone pending until its delete event lands
When a local delete raises an existing tombstone's timestamp, the new
value is only a local clock guess ahead of that delete's event. Leaving
the tombstone un-pending lets a write-back offset release it before the
event is actually consumed, reopening the resurrection window.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound tombstone replay to the tombstone TTL
Persisted-log replay retried forever, keeping lazy remote reads gated
indefinitely when the log cannot be read. Cap retries at the tombstone
TTL measured from replay start: past that point every tombstone would
have expired anyway, so opening the gate loses no protection.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: re-check deletion tombstone before persisting lazy fetch
A delete landing while StatFile is in flight passed the earlier
tombstone check but still persisted the fetched entry, resurrecting a
path whose remote delete is pending. Re-check right before CreateEntry.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: retract a lazily persisted entry when a delete raced the insert
The pre-insert tombstone check still leaves a window between the check
and the store insert. Since deletes always record the tombstone before
removing the entry, a tombstone visible right after a successful insert
means the delete already ran: delete the entry back out so the
tombstoned path stays deleted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: note why the replay deadline can safely open the gate
Deletes made after startup are captured by the live delete and event
paths, so a stalled replay can only be missing pre-restart deletes, all
of which are past the tombstone TTL by the deadline.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: retract only the entry a lazy remote read materialized
Deleting by path after a raced delete could remove a legitimate rewrite
that replaced the fetched entry. Verify the stored entry still matches
the remote object (or the just-created directory shape) before deleting,
and apply the same post-insert check to lazy listing children.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: require full-entry equality before retracting a lazy entry
Remote-only matching still removed a write that had updated the fetched
entry, e.g. appended chunks. Compare the persisted entry against what
this read materialized; any change means a real update owns the path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: VolumeMarkReadonly answers NotFound when the volume vanished under the lock
make_volume_readonly looked the volume up, notified the master (step 1),
then took the store write lock (step 2) and marked the volume only `if
let Some(..)`. When the volume left the store during step 1 -- a master
round trip, during which an unmount or a heartbeat expiry can land --
the missing else meant the RPC reported success for a volume the server
no longer has, and step 3 told the master again that it is read-only.
Go's Store.MarkVolumeReadonly (weed/storage/store.go) returns
"volume %d not found" when findVolume comes back nil, and
makeVolumeReadonly (weed/server/volume_grpc_admin.go) returns that error
before the step-3 notification. The Rust step 2 now does the same:
find_volume_mut(vid) -> Status::not_found("volume {vid} not found"), and
the `?` skips step 3, as it already did for a set_read_only_persist
failure. The scrub caller already matches NotFound to skip such a
volume instead of failing the whole report; it now actually gets it.
volume_mark_writable already returns NotFound under its write lock.
The regression test opens the step-1 window deterministically: step 1
awaits the current_master_url read lock, so the test holds its write
guard, lets make_volume_readonly park there after its own lookup
succeeded, unmounts the volume, then releases the guard. With no master
configured the notification is a no-op, so the write lock in step 2 is
the only place left that can notice the volume is gone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume server: trim comments on the vanished-volume mark-readonly path
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-25 22:05:53 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: refuse s3.bucket.create on an existing bucket
CreateEntry without o_excl replaces the bucket entry, dropping every
extended attribute: lifecycle configuration, owner, versioning and the
irreversible Object Lock flag. Send o_excl so a re-run fails with
'bucket already exists' instead of silently resetting the bucket.
* filer: fail exclusive creates when the lookup itself fails
CreateEntry discards FindEntry errors, so an o_excl create hitting a
transient store failure would take the insert path and upsert over the
entry it was meant to preserve. Propagate the lookup error when o_excl
is set; non-exclusive creates keep their existing semantics.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: test s3.bucket.create requests an exclusive create
Exercises the command end to end through a fake filer gRPC server and
asserts the OExcl flag reaches the wire along with the already-exists
error path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: synchronize captured requests and assert the exact bucket error
The fake filer records CreateEntry requests on the gRPC server goroutine,
so reads need the same mutex; the test also now checks for the exact
"bucket my-bucket already exists" message rather than any error that
mentions existence.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-25 22:04:54 +08:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: validate ext and collection in gRPC CopyFile
Port the Go-side checks (checkVolumeFileExtension, checkVolumeCollection)
to the Rust volume server so a client-supplied collection or ext carrying
a separator or ".." cannot fold a path outside the volume directory.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: validate ext and collection in gRPC ReceiveFile
Same port on the write path: the file ReceiveFile creates is built from
client-supplied fields, so reject traversal there too. Reported through
the response error field, matching Go's SendAndClose.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: stamp entries with IF_CHUNKS_EQUAL so a stale write-back cannot delete live chunks
updateLocalEntry records the RemoteEntry stamp after an upload by writing the
event's entry back with UpdateEntry. The filer deletes every stored chunk
absent from an updated entry, so when the file was rewritten while its upload
was in flight (or the event is a replay), the stale snapshot deletes the
rewrite's chunks: the entry then points at the new fid with no needle behind
it, and the rewrite's own upload fails and is skipped as superseded.
The stamp write now carries WriteCondition IF_CHUNKS_EQUAL over the event's
chunk fids, evaluated by the filer under the path lock. A refused stamp means
the filer moved past this event; the superseding event follows in the log and
stamps the current entry, so the refusal is logged and skipped like a
superseded upload.
Reproduction: weed server -filer plus a weed server -s3 remote, remote.mount,
filer.remote.sync; hold the remote (docker pause) so one upload stays in
flight, rewrite the file through the filer, unpause. Before: the entry's chunk
is 404 on every volume server. After: the stale stamp is refused, the rewrite's
chunk stays live and reads back after a vacuum.
* filer.remote.sync: stamp entries with IF_ENTRY_EQUAL so stale inline content or metadata cannot be restored
The IF_CHUNKS_EQUAL guard compared only the chunk fid multiset, so a
rewrite that touched inline content or metadata alone still compared
equal and the stale snapshot overwrote the live entry. The new clause
compares the whole stored entry against the event's entry under the
same path lock.
* filer: route conditional UpdateEntry to the entry's owner filer
Two filers locking the same path locally could still pass a stale
condition on the non-owner while the owner's entry had moved on. When a
condition or expected_extended precondition is set, forward the request
to the entry's owner the same way conditional CreateEntry does, with
is_moved bounding the hop.
* filer: compare IF_ENTRY_EQUAL against the normalized expected entry
FindEntry grows FileSize to the chunk extent, so a raw event entry with
FileSize still zero failed the condition on an unchanged file and the
stamp was skipped, letting a replay upload the object again.
* filer.remote.sync: classify refused stamps by gRPC status only
A FailedPrecondition substring in an unrelated error would have been
swallowed as a skipped stamp; status.FromError already unwraps.
* remote sync: keep the event entry intact for IF_ENTRY_EQUAL
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
OLake (github.com/datazip-inc/olake) is a CDC and ingestion engine that writes
Apache Iceberg. It covers two paths none of the existing catalog suites reach.
It is a strict Java Iceberg client. OLake does not write Iceberg from Go — its
Go process spawns a Java sidecar over gRPC and writes through the official
Apache Iceberg library, because the Go library has no equality deletes and CDC
needs them. That makes it the client class weed/s3api/iceberg's metadata
compliance backfill exists to serve: the one that fails with "Cannot parse
missing long current-snapshot-id" when spec-required keys are omitted.
And it produces equality deletes. Its upsert path commits operation=overwrite
with an equality-delete file and a delete manifest. ClickHouse, Doris, Trino,
Spark and DuckDB all only append, so nothing else in this directory exercises
a delete manifest at all.
Six subtests: the destination check reaches SUCCEEDED and actually loads the
REST catalog; discover enumerates the source; a full sync commits a snapshot;
PyIceberg reads back what the Java writer committed; an update plus a re-sync
records an overwrite carrying equality deletes with a delete manifest in the
current snapshot; and the catalog does not rewrite manifests the official Java
writer produced, which gives the manifest-repair path a negative test to go
with the ClickHouse positive one.
What it deliberately does not assert is a delete-applied read. PyIceberg
refuses to scan a table carrying equality deletes (apache/iceberg#6568) while
reading its metadata fine, and an engine that can apply them costs a
multi-gigabyte image. Recording the commit correctly is the catalog's
contract; applying deletes on read is the engine's. The README says so, says
the read half was verified by hand once with StarRocks, and warns against
later "upgrading" this to a PyIceberg rows read — which would either fail or,
if PyIceberg ever starts skipping deletes instead of raising, pass by not
looking.
The workflow job asserts the suite actually ran rather than trusting a green
exit, for the same reason: at least one top-level PASS and zero SKIP.
No product change — the destination config is the generic catalog_type=rest
with the standard OAuth2 client-credentials flow, and s3_path_style is not even
set, since OLake turns it on itself whenever s3_endpoint is non-empty.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* filer: resolve the collection a bucket delete drops
A bucket delete dropped the collection named after the bucket, which
assumes bucket name is collection name. With a collection rule the
write path honors, deleting the bucket either orphaned its collection
or, when a bucket was named after a shared collection, removed volumes
other buckets still write to.
Resolve the collection through the same rule chain the write path uses
and drop it only when no other bucket resolves there too. A listing
failure keeps the collection, the safe side of an unknown.
* filer: prove collection exclusivity across all paths before dropping it
The sibling-bucket scan missed every non-bucket writer: a broad rule like
'/' or '/buckets/', a rule under a surviving bucket, or a rule on an
unrelated path can route into the same collection. Check every storage
rule's prefix instead, and mirror the grouped gateway's explicit
<group>_<bucket> collection, which otherwise resolves a rule-named
collection the bucket never wrote to.
* s3: let the filer own the collection decision on bucket delete
Both entry points deleted a name-derived collection around the filer's
own resolved delete, bypassing its exclusivity check and wiping sibling
data. The filer now resolves the collection a bucket actually used,
including the grouped form.
* filer: keep a collection the default write route also uses
Rule-less writes outside buckets land in the filer's default collection,
so a bucket resolving there shares it with them.
* topology: wake the vacuum dispatcher when a worker frees quota
The dispatch loop slept a fixed 10s whenever every pending volume was
waiting for a per-server quota slot, so a sweep took volumes x 10s
regardless of how fast the compactions were. Workers now signal on a
buffered channel after crediting quota; the dispatcher waits on it with
the 10s sleep kept only as a timeout.
* master: add -vacuumIntervalSeconds to tune the automatic sweep interval
The 14-minute base interval was a literal inside the refresh loop while
every neighbouring vacuum knob was already a flag. Defaults to 840s,
unchanged.
* topology: keep the 14 minute floor on the vacuum interval
A zero-valued MasterOption or a negative -vacuumIntervalSeconds left the
sweep sleeping only its jitter, so treat non-positive intervals as the
previous default.
The parity pass only queued shards past the per-type cap, so a single
parity shard sharing a rack with data was never a move candidate even
when an empty data-free rack existed (2+1 over 3 DCs settled 2/1/0).
Non-overflow candidates now move too, but only to a rack without data;
overflow shards keep the existing data-rack fallback.
* volume server: sweep stale EC artifacts before VolumeEcShardsGenerate re-encodes
The Rust VolumeEcShardsGenerate went straight into write_ec_files: no unload
of an already-mounted EC volume and no stale-artifact sweep. Only .ec00..ecNN
on the encoding disk were truncated, so a retry could mix two encode runs. A
stale N.ec03 left on a sibling disk survived, reconcile later mounted it
against the new .ecx, and the new .vif made the encode_ts_ns identity guard
pass, so reads served old-run bytes at new-run offsets.
Mirror Go's VolumeEcShardsGenerate (#9880 / #9953): UnloadEcVolume on every
disk, then removeStaleEcArtifacts on every disk location before encoding.
remove_ec_volume_files_full_teardown already has removeStaleEcArtifacts'
semantics (.ec00..ec31, .ecx/.ecj/.ecsum[.vN] in both the data and idx dirs,
.vif only on a shard-only disk; never the source .dat/.idx), so reuse it. Add
Store::unload_ec_volume, which unlike remove_ec_volume does not stop at the
first disk and closes the descriptors so the unlink frees the inodes. The
store write lock covers only unload + sweep, not the encode.
The failure arm now also drops the generation-0 .ecsum, as Go's defer does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume server: wake the heartbeat after VolumeEcShardsGenerate unloads shards
The pre-encode unload drops mounted EC shards from memory, but unlike every
other unmount path it did not wake the heartbeat, so the master kept routing
reads to shards this server no longer serves until the next pulse. Notify
once the store lock is released, and before the sweep error propagates: a
failed sweep has unloaded the shards too.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume server: clean up encode artifacts when the .vif write fails too
Go's shouldCleanup defer covers every error before the .vif commits,
not just a failed encode. A serialize or write failure on the .vif left
the fresh .ecNN/.ecx/.ecsum behind, which the next generate would have
to rely on the new sweep to remove. Extract the cleanup and run it on
the .vif error paths as well.
* volume server: write the EC .vif atomically
Go's SaveVolumeInfo writes a temp file, syncs it, and renames it over
the target, so a failed write leaves the previous metadata intact and a
read-only .vif fails the save. The direct fs::write truncated the file
first, so a write or sync failure could leave an empty .vif even after
cleanup_encode removed the generated shards.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: an EC volume needs a non-empty .ecx to mount
Two gaps against Go in how the Rust volume server treats the .ecx.
EcVolume::new mounted with no index at all. The per-shard
VolumeEcShardsMount path picks the disk by shard file alone, so a shard
whose .ecx was on no local directory still registered and was
advertised to the master; every VolumeEcShardRead then failed with
"ecx file not open", and add_shard's 0-byte guard was neutralised
because ecx_file_size stayed 0. Go's NewEcVolume returns an error
wrapping os.ErrNotExist. EcVolume::new now fails with NotFound, and
Store::mount_ec_shard looks up the .ecx owner across all disks first
(findEcxIdxDirForVolume) so a shard on a sibling disk of its index
still mounts instead of turning into a hard failure.
A 0-byte .ecx stub, as left by a failed EC distribute copy, counted as
a valid index. Go requires Size() > 0 wherever the file steers a
decision: HasEcxFileOnDisk, findEcxIdxDirForVolume, indexEcxOwners
(shared by reconcile and mirror), and VolumeEcShardsCopy removes a
copied 0-byte .ecx and fails the copy. Mirror each through one
is_usable_ecx_file helper. NewEcVolume itself still accepts a lone
0-byte .ecx as a legitimate empty index, but prefers a non-empty copy,
local directory first, over a stub in the other directory; the
resolution in EcVolume::new now follows the same order.
Tests that mounted EC volumes without any .ecx get a real fixture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: mount_ec_shard tries every disk; reconcile ignores a 0-byte local .ecx
mount_ec_shard returned the first disk's error, so an unusable shard copy
(a 0-byte .ecNN left by an interrupted move) hid a good copy on the next
disk. Like Go's MountEcShards, keep scanning: NotFound means "not this
disk", any other failure is collected, and an all-disks-fail error names
every disk tried. "No .ecx on any local disk" is now told apart from
"shard not on this server".
The orphan-shard reconcile took its locally-mirrored fast path whenever a
local .ecx existed at all. A 0-byte stub there registered the shards against
an empty index while the owner index skipped that same stub. Go gates the
fast path on HasEcxFileOnDisk; do the same. ec_local_ecx_path loses its last
production caller and becomes test-only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: match Go's mount error text and skip the owner stat on the owning disk
MountEcShards in Go skips the HasEcxFileOnDisk stat when the disk's own
directories already hold the .ecx, dedups a shared -dir.idx across
locations in findEcxIdxDirForVolume, and reports "load failures" with
the same wording. Also drop two issue-number references from comments.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
With SEAWEED_WRITE_QUEUE=1 every upload came back with ETag "00000000".
The upload handler built the needle with Needle::default(), so its
checksum was CRC(0), and handed a clone of it to the queue. The CRC was
only computed in the write path, on the worker's clone, and WriteResult
carries no checksum back, so n.etag() in the handler formatted the zero
checksum. The direct path writes through &mut n and was correct.
Compute the checksum in the handler while building the needle, the way
Go's CreateNeedleFromRequest does, over the same bytes the write path
hashes (the stored data, gzipped or not). The ETag and the has-name flag
are read before the write, so the needle is moved into the queue instead
of cloned, which also drops a full payload copy per queued upload.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* volume: walk_index_file keeps row alignment across short reads
walk_index_file issued one Read::read per batch and decoded whatever came
back. Read::read may legally return a short count that is not a multiple
of the 17-byte entry size (FUSE and network filesystems, a BufReader whose
capacity is not a multiple of 17). The split entry at the end of the batch
was dropped with no carry and the next read started mid-entry, so every
later row was decoded from misaligned bytes and fed to the index as a
garbage key/offset/size. This function backs every in-memory index load.
Go's WalkIndexFile is immune because it reads through io.ReaderAt, which
returns a full buffer or an error. Fill the batch buffer until it is full
or the reader reports EOF, retrying ErrorKind::Interrupted, and only then
decode whole entries. Reads stay batched at ROWS_TO_READ entries.
EOF semantics are unchanged and match Go: on io.EOF Go decodes the whole
entries in the final buffer, ignores a trailing partial entry and returns
nil. A torn final entry is still skipped without an error here.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume: trim walk_index_file comments
The batch-fill loop and the ShortReader test helper each carried a
paragraph where a sentence suffices.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* lance: accept OAuth2 bearer tokens for catalog auth
Lance and LanceDB clients can only send OAuth2 / Bearer / API-Key
headers on catalog calls, never SigV4, so behind an auth-enabled S3
gateway every namespace request failed with 403 Access Denied.
Mirror the Iceberg catalog's OAuth2 support: POST /oauth/token accepts
an S3 access key / secret key as client_id / client_secret, validates
them against IAM, and returns a signed JWT. The Auth middleware accepts
that token as a Bearer credential before falling through to SigV4.
Closes#11430
* lance: accept x-api-key header carrying an S3 credential
The Lance namespace spec's third auth scheme maps api_key onto the
x-api-key header. Accept "access_key:secret_key" there and validate it
against IAM, so clients that only hold static headers can authenticate
without minting a token first.
* lance: answer invalid_client with the Basic challenge
RFC 6749 5.2 requires a 401 from the token endpoint to carry
WWW-Authenticate matching the scheme the client used, so it knows how
to retry.
* lance: cap the token endpoint request body
/oauth/token is unauthenticated, so ParseForm needs the same size
bound decodeBody applies to every other catalog request.
* lance: keep query strings out of request logs
/oauth/token rejects a client_secret sent in the query, but the
logging middleware and the catch-all wrote RequestURI to the log
before that rejection ran. Log the path alone so a mis-sent secret
never reaches the log.
* lance: log the escaped path, not the decoded one
URL.Path decodes percent escapes, so a request like /%0aFORGED could
split log lines. EscapedPath keeps the encoding while still dropping
the query string.
* fix 11400
* persist failed-recovery quarantine and harden rollback
- record the unavailable state in a .unavailable marker, fsync it, and
re-arm it on load so a restart cannot serve an unverified pair
- quarantine the volume so heartbeats stop advertising it
- block MarkVolumeWritable while unavailable, rechecked under noWriteLock
- fail every request of a failed batch, not only the succeeded ones
- restore the needle map and truncate .dat on inline fsync rollback failure
- add truncateIndex for the sorted-file needle map
- mirror the fail-closed semantics in the Rust volume server
* volume: erase rolled-back mappings instead of leaving tombstones
A rolled-back batch or failed inline write used Delete() to undo a
needle that did not exist beforehand, leaving a tombstoned map entry
whose stale offset makes the next write to that needle fail reading a
header that no longer exists. Add removeMapping/restoreMapping to the
mappers so recovery erases entries that were absent before the batch
and reinstates the exact prior offset/size for ones that were,
including tombstones. The index row still goes through Delete so a
replay forgets the needle.
* volume: gate bulk readers on unavailable and fsync the marker's dir
- fsync_dir(&self.dir) synced the volume dir's parent, not the dir
holding .unavailable; pass the marker path so the create survives
a host crash
- export UnavailableError and check it in ReadAllNeedles,
VolumeTailSender, VolumeIncrementalCopy, and IncrementalBackup so
replica-sync paths cannot stream or append data from an unverified
.dat/.idx pair; mirror on the Rust side via read_dat_slice,
read_all_needles, dat_scan_plan, and the incremental-copy handler
* volume: drop issue references from comments near touched code
* volume: stop active scans when the volume becomes unavailable
The stream entry-point checks ran once per RPC, so a volume quarantined
by a failed recovery mid-scan kept serving data. Recheck availability
per needle/chunk on the detached read paths: tail scan and heartbeat,
read-all, incremental copy, incremental backup writes, and the Rust
StreamingBody chunk reads. Rust incremental copy also rejects a
quarantined volume before sync_to_disk touches the backend.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust volume: test makeup_diff replay across a 32 GiB offset boundary
Issue #11410 corrupted a replayed write's index offset in Go's makeupDiff
by patching only four of the five offset bytes. The Rust makeup_diff
already encodes the whole offset through idx_entry_to_bytes and
Offset::from_actual_offset; this adds the mirror of
TestConcurrentWriteCrossesOffsetBoundary so a regression would fail here
the same way it does under -tags=5BytesOffset on the Go side.
Sparse-truncate the .dat to 64 GiB, compact, write, commit: the index
offset must equal the .cpd size and the needle must stay readable
through a second vacuum. Gated on the 5bytes feature since a 64 GiB
.dat exceeds the 32 GiB range of 4-byte offsets.
* rust volume: skip the offset-boundary replay test on Windows
Windows set_len allocates the full 64 GiB extension instead of a sparse
range, so the test fails with StorageFull on CI runners. Gate it to unix,
where set_len leaves the extension unallocated.
* volume: encode all offset bytes when makeupDiff replays a write
makeupDiff patched only bytes 8:12 of the index entry, so under the
5BytesOffset build the fifth byte kept the old offset's high bits and the
replayed needle's index pointed 32 GiB-aligned ranges away from its body.
A later vacuum then dropped the entry as unreadable. Rebuild the entry with
needle_map.ToBytes, the same encoder the tombstone branch just below uses.
* volume: test makeupDiff replay across a 32 GiB offset boundary
Sparse-file test: truncate the .dat to 64 GiB after one write, compact,
write a second needle, commit, and assert the index offset matches the
.compacted size and the needle stays readable through a second vacuum.
Only runs under -tags=5BytesOffset.
* volume server: reject non-ASCII input instead of panicking
Three parsers sliced attacker-supplied strings by byte offset, so a
multi-byte character split inside itself and panicked the task:
- parse_needle_id_cookie took the last 8 bytes as the cookie and the
rest as the needle id. Reachable from VolumeServer.BatchDelete,
whose file_ids come straight off the wire as protobuf strings;
that handler already answers 400 per bad fid, so the guard turns a
panicked RPC into the error it was already written to return.
- TTL::read took the unit as the last byte and the count as
everything before it, so "?ttl=5<multi-byte>" split mid-character.
The HTTP upload path does TTL::read(..).ok() and drops an invalid
TTL; AllocateVolume maps the Err to InvalidArgument.
Both now reject non-ASCII up front. Hex and a digits-plus-unit TTL are
ASCII by definition, so no accepted input changes -- covered by tests
alongside the rejection cases.
The six response-* header overrides were inserted with
parse().unwrap(). They come from the query string, so
"?response-cache-control=%0Aevil" decodes to a value HeaderValue
rejects and the unwrap panicked the connection task,
unauthenticated. They now skip the override, matching the if-let the
chunked-response path in the same file already uses.
ReplicaPlacement::from_string was reported as a fourth site but is not
one: reaching chars[2] requires chars[0] and chars[1] to be ASCII
digits, which forces the padded string to be three single-byte
characters, so a multi-byte character always lands on a to_digit()
None first. Kept as a regression test rather than a change.
Each fix was confirmed against the unfixed code first: the parser
tests panic with "byte index N is not a char boundary", and the
integration tests panic at handlers.rs:1413 and ttl.rs:88.
Not a vector, contrary to the report: the HTTP request line. The path
is not percent-decoded before parsing, so "%C3%A9" stays ASCII and
fails the length check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* volume server: fall back to needle MIME when response-content-type is invalid
Skipping an unparseable override left the response without any
Content-Type because the override had already bypassed the normal MIME
selection. Also correct a test comment that described a chars[2] panic
which cannot be reached.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* s3: count seaweedfs-quota as an operation subresource
PUT /bucket?policy&seaweedfs-quota was not rejected by
hasAmbiguousSubresource because operationSubresources omitted the
seaweedfs-quota key. The router then picks the policy route
(registered first) while the IAM action resolver may resolve the
request to s3:PutBucketQuota, letting a quota-only identity write a
bucket policy. Reject the combination before routing, matching the
fix for policy&tagging (#10987).
* s3: resolve seaweedfs-quota after other bucket subresources
The quota routes are registered last among the bucket subresource
routes, but the action resolver found seaweedfs-quota inside the
unordered bucketQueryActions map, so a request carrying it alongside
another selector could be authorized as the quota operation while the
router served the earlier-registered handler. Resolve it explicitly at
the end so the resolver agrees with the router, mirroring how
list-type is handled.
* s3: count resolver subresources in the ambiguity guard
hasAmbiguousSubresource only counted operationSubresources, so adding
a query parameter to the action resolver without updating that list
reopened the authorize-one-serve-another gap. Count bucketQueryActions
keys as operation selectors too, and add a test that walks the
registered routes and fails on any query key that is neither an
operation subresource nor a known modifier.
* volume: load the .ecj deletion journal in chunks, and repair a torn tail
Two independent defects in the EC deletion journal's load path.
1. The loader issued one NEEDLE_ID_SIZE-byte positional read per entry.
That is fine for a healthy journal -- kilobytes -- and pathological for a
large one. A `.ecj` is semantically a SET of deleted needle ids but is
written as an append-only log that nothing dedupes, and several paths append
a peer's ENTIRE journal onto the local one (VolumeEcShardsCopy with
copy_ecj_file, EC index recovery, and ec_decode's deliberate cross-holder
merge), so a volume whose shards are repeatedly balanced between two servers
grows the file without bound.
Observed in production: 1.51 TB and 1.30 TB on the two holders of one 10+4
volume containing ~100 distinct ids. At that size the per-entry loop is
~188e9 syscalls, run synchronously while holding the deleted_needles write
lock and before the HTTP port opens. The process sits at 100% of one core
with a small RSS -- the set stays tiny because the ids repeat -- reading at a
few MiB/s because 8-byte reads defeat readahead, logs nothing after "Adding
storage location", and ignores SIGTERM. The master then unregisters every
volume it holds and reads of them fail. 4.46 and 4.47 are both affected.
Read in 1 MiB chunks and build into a local set, merging once at the end so
the write lock is not held for the whole scan. Measured on a 256 MiB journal
of 100 distinct ids: 33,554,500 syscalls -> 257, identical resulting set.
2. A torn tail silently corrupted later deletes.
The journal handle is in append mode, so writes land at the physical end
regardless of alignment. A trailing partial record therefore pushed every
later append out of alignment: the loader skipped the partial bytes, but the
next mount decoded them together with the leading bytes of the following
entry, producing one garbage id and dropping the delete that came after the
tear -- after acknowledging it.
Truncate to a whole number of records at mount, before anything can append.
The repair uses its own read+write (non-append) handle: on Windows,
append(true) requests FILE_APPEND_DATA without FILE_WRITE_DATA (and
.write(true) is subsumed by .append(true)), so SetEndOfFile through the
journal handle fails with ERROR_ACCESS_DENIED.
The same trap exists in journal_delete's recovery path, which calls set_len
on the append handle to roll back a partial write whose sync failed. It is
error-handled rather than fatal, so on Windows that rollback silently does
not happen. Untouched here; worth a separate fix.
Bounding the journal's growth needs compaction, which is deliberately not in
this change: replacing the file under a store that can hold several EcVolume
instances for one volume id requires coordinating with the other holders, and
that belongs at the store layer. Sent separately.
Tests: a journal spanning several read chunks loads every entry; a trailing
partial record is ignored rather than panicking; a torn tail is truncated at
mount and a delete taken afterwards survives a remount.
* volume: roll back a failed .ecj append through a dedicated write handle
The append handle lacks FILE_WRITE_DATA on Windows, so the set_len
rollback after a failed sync silently did nothing and the journal could
drift one record past deleted_needles. Same trap as the torn-tail repair
in this file; fix it the same way. Also format the new tests.
* volume: mirror chunked .ecj load and torn-tail repair in Go
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
* volume server: ReceiveFile loses bytes and hides fsync failures
Three defects in one handler, all on the path that receives a pushed
.dat/.idx/.vif or EC shard:
- `f.write(&content)` never compared the return to content.len().
A short write (ENOSPC, NFS) counted only the bytes that landed,
so every later chunk was written at a shifted offset and the RPC
answered error: "" with a byte count that looked right. Go's
os.File.Write loops. Now write_all.
- `let _ = f.sync_all();` discarded EIO and answered success with
the full byte count. Go omits the check too, but
ReceiveFileResponse carries an `error` field and the caller
renames the staged file into place on success -- so a silent
fsync failure publishes a file whose data never reached the
platter. Flush and fsync failures are now reported.
- Both the per-chunk write and the final fsync were blocking
std::fs calls inside the async fn, on the runtime worker that is
also driving the stream. Switched to tokio::fs + BufWriter, the
shape `drain_copy_stream_to_file` in this same file already uses
and documents. The partial-file cleanup on the error path moves
to tokio::fs::remove_file for the same reason.
The handler had no test at all, which is how the short-write bug
survived. Added a round-trip over a real connection with ragged chunk
boundaries, asserting the bytes on disk and not only the reported
count -- a dropped or reordered chunk changes the file even when
bytes_written still adds up.
That test guards the rewrite; it does not reproduce the original
faults. ENOSPC and EIO need fault injection that this suite has no
harness for, so the short-write and fsync paths are argued from the
code, not demonstrated by a failing test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* volume server: remove the staged file on every ReceiveFile error reply
Flush and fsync failures returned early and left the partial .copying or
shard file behind, as did the pre-existing write-error path. Route all
response-level errors through one cleanup block, matching Go's
close-and-remove on a failed write.
* volume server: tighten ReceiveFile comments
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
* volume server: HTTP DELETE on a distributed EC volume
The delete handler validated the cookie with EcVolume::read_ec_shard_needle,
which reads only locally-mounted shards and errors "ec shard N not available
locally" for any interval held by a peer. Every Err was mapped to 500 and no
.ecj tombstone was appended, so on a standard 10+4 spread over 14 servers an
HTTP delete of an EC needle could not succeed. The GET path already goes
through read_ec_shard_needle_distributed.
Route the delete's read through the same distributed reader. It does a
local-first pass in its snapshot phase, so the all-shards-local case costs
what it did before, and no store guard is held across the await (the reader
takes its own; RwLockReadGuard is !Send).
Two smaller corrections fall out of the new return type:
- the reader reports both "needle not in the index" and "volume vanished
between the has_ec check and the snapshot" as Ok(None), which collapses
the old Some(Ok(None)) and None arms into one 404;
- an io::ErrorKind::NotFound now answers 404 rather than 500, matching the
GET path. Telling a caller to retry a delete that can never succeed was
half the bug.
The cookie check and its ordering before the journal append are unchanged.
Not addressed here: Rust journals the tombstone locally while Go routes it to
the primary shard holder. That is a separate behaviour change and belongs in
its own PR against the same issue-10 checkbox.
The regression test mounts 13 of 14 shards, leaving out the one holding the
needle's interval. The distributed reader seeds its Reed-Solomon buffers from
locally mounted siblings, so with >= 10 survivors it reconstructs with no peer
fan-out -- which makes the bug reproducible on a single node. Against the
unfixed handler the test fails with 500 vs 202.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* volume server: fail the delete when the EC volume unmounts mid-request
find_ec_volume_mut returning None used to fall through to a 202 with no
.ecj tombstone written, reporting success for a delete that did not
happen. Answer 404 like the other volume-vanished arms so the caller can
retry after a remount.
* volume server: forward EC needle deletes to a primary-shard holder
Mirror Go's doDeleteNeedleFromAtLeastOneRemoteEcShards: the tombstone is
journaled on one holder of the needle's primary data shard via
VolumeEcBlobDelete (or the local journal when this server holds the
shard), falling back to any other shard holder when the primary has
none. Journaling only on the node that received the DELETE scattered
tombstones across whichever server took the request.
* volume server: route BatchDelete EC deletes through the same forwarding
BatchDelete had the same local-journal divergence as HTTP DELETE, plus a
gap the old code admitted in a comment: the .ecx index cannot supply the
needle's cookie, so EC deletes ran with no cookie check at all. A
distributed read now fills the needle for every EC entry — matching Go's
DeleteEcShardNeedle, which reads and compares the fid cookie even when
skip_cookie_check is set — and the tombstone forwards via
delete_ec_shard_needle_distributed. A needle deleted between read and
journal reports 304 like Go's ErrorDeleted; a vanished volume reports
500 so the filer retries.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
server/grpc_client.rs stopped at build_grpc_endpoint() -> Endpoint, so all 13
production call sites hand-wrote the same .connect() + X::with_interceptor()
+ two max_*_message_size() lines. Four of them -- VolumeCopy,
VolumeTailReceiver, VolumeEcShardsCopy and the HTTP chunk batch-delete fan-out
-- dialed with no timeout at all, so an unreachable peer whose TCP handshake
never completes (SYN dropped, blackholed route, host behind a silent firewall)
left the operation waiting on the kernel's own retry budget, minutes long.
Add GrpcDialOptions (unary / long / stream presets), connect_channel(), and
volume_server_client() / master_client() / filer_client() constructors that
attach the request-id interceptor and lift both message-size limits, then
route all 13 sites through them. build_grpc_endpoint is private again, so
connect_channel is the only way out of the module and no call site can dial
without picking up a bound. Each site's existing timeouts are preserved
exactly; the four bare dials gain a 5 s connect timeout and nothing else. No
per-request deadline was added to any streaming call: Endpoint::timeout is a
per-request bound on time-to-first-response-headers for every request the
channel carries, so a value picked for one short call would also be the header
deadline for the whole-volume transfer sharing the dial.
The new bound covers the TCP handshake only -- tonic hands connect_timeout to
HttpConnector::set_connect_timeout. A peer that completes the handshake and
then stalls in the TLS or HTTP/2 exchange is still unbounded at those four
sites, as are the RPCs themselves. That is why the three ping_* helpers keep
their outer tokio::time::timeout: replacing it with connect_timeout would have
narrowed a whole-connect bound they already had.
main.rs no longer re-declares GRPC_MAX_MESSAGE_SIZE and the three
keepalive/window constants; it imports them from grpc_client.rs so the
inbound server and the outgoing clients cannot drift apart.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: share the I/O-error tracker between Volume and EcVolume
Volume and EcVolume each carried the same three fields - a mutex-held
last error, a consecutive count and a sticky quarantine flag - and the
same four methods over them, identical except for the path qualifier on
is_storage_io_error. The tolerance the count is compared against was a
fourth copy: heartbeat.rs held VOLUME_IO_ERROR_TOLERANCE for volumes,
ec_volume.rs held IO_ERROR_TOLERANCE for EC, and the volume test helper
open-coded the same 3, so the two paths could drift apart silently.
Go keeps this in one place already: weed/storage/io_error.go holds
IoErrorTracker, IoErrorTolerance and isStorageIoError, and Volume embeds
the tracker. Go's EcVolume has to re-implement it only because those
fields are unexported and EC lives in another package.
storage::io_error::IoErrorTracker now owns that state, with record /
state / should_quarantine / mark_quarantined / reset and the single
IO_ERROR_TOLERANCE. is_storage_io_error moves into the same file, so it
sits with the tracker that is now its only caller, the way io_error.go
is laid out. Both volume kinds embed one tracker and keep their existing
method names as delegates, so the ~16 internal call sites and the
readers in heartbeat.rs, store.rs and grpc_server.rs change only where
the two threshold comparisons become should_quarantine().
Volume::last_io_error and EcVolume::reset_io_error_state had no callers
and are gone.
Unchanged: what counts as a storage-media error - is_storage_io_error
changed file, not body, and is still the single predicate both volume
kinds share, where Go's EcVolume tests EIO directly and so misses the
Windows codes. Also unchanged: the tolerance value, the metric increment
on every counted error, and the sticky quarantine - a success clears the
count and the last error but never the flag, which only reset lifts. In
the heartbeat the state read moved inside the quarantine branch, so the
common path no longer takes the tracker's mutex or clones the last-error
string; should_quarantine's two relaxed loads run either way.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: hoist absolute_display_path into server
handlers.rs and ui.rs each held a byte-identical copy of the helper that
turns a configured -dir into an absolute path for display. The status
JSON and the status page are meant to show the same directory, so the
two copies had to be edited together to stay that way.
The helper now lives in server/mod.rs as pub(crate) and both callers use
it. No behaviour change: same body, same call sites.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: keep EcVolume::reset_io_error_state
Moving both volume types onto the shared IoErrorTracker dropped
EcVolume's public reset while Volume kept its own, so the two sides of
the tracker drifted apart.
mark_quarantined is sticky: a later successful read clears the error
count through record(), but the quarantine flag only comes down through
reset(). Without the delegate an EC volume that hit sustained media
errors could not be returned to service in place once the storage was
repaired. Go exposes the same method as EcVolume.ResetIoErrorState
(weed/storage/erasure_coding/ec_volume.go:114).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* rust volume: name the shared tracker after Go's IoErrorTracker
- check_read_write_error, get_io_error_state, mark_io_quarantined,
reset_io_error_state match weed/storage/io_error.go one to one
- io_error module is pub(crate) like the io module beside it
- restore EcVolume::reset_io_error_state so both volume kinds expose the
same recovery surface
- trim comments that restate the code
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust: a seaweed-common crate for the address and TLS helpers both crates carry
seaweed-volume and seaweed-worker are separate cargo trees with separate
lockfiles and no root manifest, so anything both of them need has had to be
written twice. Two of those copies are a correctness risk rather than a typing
cost, and this crate is where they stop being copies.
address.rs is the HTTP<->gRPC port rule: `host:port` means gRPC on port+10000,
`host:port.grpcPort` names it outright. The two copies had already drifted —
the worker's bracketed IPv6 literals, the volume server's did not — so the rule
lives here once, returning a typed AddressError whose Display text is the volume
server's original wording, with join_host_port public beside it. A test asserts
two of those messages in full rather than by substring, because the wording is
the contract its callers hand to a Status or an io::Error; the other three end
in a std ParseIntError message, which is std's to reword. The enum is
#[non_exhaustive] so a future variant is not a breaking change for either
consumer. The tests are both crates' cases together, plus the IPv6,
already-bracketed and normalisation cases neither copy covered on its own.
tls.rs is install_default_crypto_provider. Both binaries link aws-lc-rs and ring
transitively, so rustls cannot auto-select and tonic's client TLS panics on
first use; each binary has to pin one and it has to be the same one, which is
exactly the kind of choice that should not exist twice. It is safe to share
because `cargo tree -i rustls` resolves a single rustls in each tree (0.23.37 in
seaweed-volume, 0.23.43 in seaweed-worker) and cargo unifies all
semver-compatible `rustls = "0.23"` requirements into one crate per binary, so
this crate writes the same process-wide static its consumer reads. rustls is
already in both graphs — directly in the volume server, through tonic's
tls-aws-lc in seaweed-worker-core — so the dependency adds no crate to either.
rust-version is 1.91.1, the lower of the two consumers' floors, so depending on
this crate cannot raise either tree's MSRV; verified with
`cargo +1.91.1 check --all-targets`. The lockfile is committed even though this
is a library: CI builds it directly, so a committed lock is what makes those
runs reproducible and their caches stable.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust: take the address and TLS helpers from seaweed-common
Both public signatures are kept, so no caller outside the two wrapper files
changes. parse_grpc_address stays `Result<String, String>` and maps the typed
error through Display; server_to_grpc_address stays `Option<String>` and drops
it with .ok(). Their doc comments and the volume server's 13 call sites are
otherwise untouched.
Three behaviours change, each in the direction of the copy that was already
right:
- The volume server now brackets IPv6 literals. `::1:19333` used to come back as
`::1:29333`, which build_grpc_endpoint rejects with "invalid gRPC endpoint
http://::1:19333: invalid authority" — an IPv6 master or EC peer could not be
dialled at all. Two tests in grpc_client.rs pin it, one on the string and one
on the endpoint the string builds.
- The volume server now emits the *parsed* gRPC port of the dotted form instead
of the original text it had just validated, so `host:8080.018080` and
`host:8080.+18080` come back as `host:18080` rather than as authorities the
URI parser rejects. Same port either way; only malformed spellings change.
- The worker's dotted form now validates the HTTP port it discards.
`server_to_grpc_address("host:abc.18080")` used to answer Some("host:18080");
it now answers None, which is what the volume server's copy has always done.
install_default_crypto_provider becomes a re-export in both trees, so
`crate::security::tls::install_default_crypto_provider` and
`weed_lance_worker::tls::install_default_crypto_provider` still resolve. The
lance crate's `rustls = "0.23"` was its only direct use of rustls and goes away
with the body; seaweed-common states the same requirement, so neither the
resolved version nor the enabled features move in either lockfile.
The PEM test fixtures stay where they are. The two tests that use them are not
duplicates: the volume server's exercises build_grpc_endpoint, and the lance one
exists precisely because aws-lc-rs and ring are both linked in that crate's
graph. Only the literals are shared, and exporting test fixtures from a library
to dedupe two constants costs more than it saves.
A path dependency outside both trees means every build context that copies one
crate directory has to copy the other. The repo has one: the Rust source-build
stage of docker/Dockerfile.go_build, which now copies seaweed-common beside
seaweed-volume. Every workflow whose `paths:` filter keys on a crate directory
gains `seaweed-common/**` — the two Rust test workflows, rust_binaries_dev,
container_dev and performance. The tag- and dispatch-triggered ones
(rust_binaries_release, container_release_unified, container_latest) have no
`paths:` filter and need nothing.
The two Rust test workflows also run `cargo test` in seaweed-common, from their
unit-test job, because a path dependency is not a workspace member and neither
tree's own `cargo test` reaches it. Each step builds into its job's cached
target directory, and both cache keys now hash seaweed-common/Cargo.lock as well
so a change there invalidates the cache it would otherwise silently reuse.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docker: keep go_build working for BRANCH revisions without seaweed-common
The rust_builder stage copies seaweed-common unconditionally now that seaweed-volume path-depends on it, but BRANCH can name any revision — including ones that predate the crate. Create the directory in the builder stage so the COPY always has a source; an empty dir beside an old seaweed-volume is harmless.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust volume: one S3 tier registry instead of two kept in sync by hand
`VolumeServerState.s3_tier_registry` and `global_s3_tier_registry()` held
the same S3 tier backends. `apply_storage_backends` — the only production
writer — registered every backend into both, and each half of the tiering
code then read a different one: the gRPC tier-move handlers resolved the
backend from the per-server field, while `Volume`'s remote mount and
destroy paths resolved it from the global registry, because a `Volume` has
no handle to the server state. Two registries that must agree, kept in
agreement by a duplicated `register_s3_backend` call and a comment in a
test constructor explaining the hand-sync.
Delete the field and let both tier-move handlers resolve from the global
registry, so `apply_storage_backends` registers once and no longer needs
the server state at all. Injecting a registry handle through `VolumeSpec`
instead was considered and rejected here: it would touch every `Volume`
constructor for no functional gain, and the process-wide registry is what
`Volume` already uses.
Behaviour is unchanged: the same names were registered in both registries,
so every lookup resolves exactly as before. The tier-down test now
registers its backend only in the global registry — before this change it
fails with `remote storage s3.tier_down_delete not found from supported:
[]`. The tier-up handler had no test at all, so it gets a cheap probe:
register a backend only in the global registry, ask for that destination,
and check the call gets past the lookup — the response is dropped straight
away, so the transfer sees a departed caller and never opens a connection.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: await the tier-up probe terminal error instead of racing it
Dropping the response left it to chance whether the detached transfer saw the closed channel before its initial check; if it won that race it went on to attempt the multipart upload with no one waiting on the outcome. Hold the stream and read until the dead endpoint fails the upload — the terminal error proves the task ran and finished, so no background network work outlives the test.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The per-EcVolume shard-location cache was three fields under three locks:
an RwLock<HashMap> for the map, a Mutex<Option<Instant>> for the time it
was last refreshed, and a Mutex<bool> for the stale mark. Nothing tied
them together. merge_shard_locations published the merged map, released
the write lock, and only then stamped the refresh time; both readers
(scrub_ec_volume_distributed's snapshot and build_snapshot) took the two
guards one after the other. A reader landing between the two writes
paired a freshly merged map with the previous lookup's timestamp -- and
that pair is exactly what needs_refresh judges, so a read went back to
the master for a map that had just been refreshed. Go keeps the same
state in one struct behind one ShardLocationsLock. replace_shard_locations
documented itself as "a single observable step" while being two.
Fold the three fields into one ShardLocationCache behind a single RwLock.
merge_shard_locations upserts and stamps in one write section,
shard_locations_snapshot returns the map and its time from one read
section, and mark_shard_locations_stale / claim_shard_locations_refresh
move the mark's read-and-consume onto the cache. The three zero-caller
accessors -- set_shard_locations, replace_shard_locations,
get_shard_locations -- are deleted, and the field is now private, so the
invariant cannot be sidestepped from outside the module. The two test
seeding sites go through merge_shard_locations, which already produces
the state they were writing by hand.
Unchanged: the freshness rule. needs_refresh keeps its thresholds and
still judges the caller's snapshot -- the map that caller will actually
read from, not whatever is cached by the time the claim runs -- so only
the stale mark is read from under the new lock. The master lookup, the
completeness guard in write_back_shard_locations and the per-shard upsert
semantics are untouched.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: typed errors for store compaction so gRPC can answer NotFound
The vacuum entry points on `Store` returned `Result<_, String>`, so the
gRPC layer had nothing to branch on and answered `Status::internal` for
every failure. A vacuum loop that races a volume being moved or deleted
saw the same code as a disk going bad, and `weed shell` could only tell
the two apart by matching on the message text.
`VolumeError` gains `VolumeNotFound(VolumeId)` — the existing `NotFound`
is needle-level and carries no payload — and `InsufficientSpace`, and
`compact_volume`, `commit_compact_volume`, `cleanup_compact_volume` and
`delete_collection` return it. `impl From<VolumeError> for tonic::Status`
in `server/mod.rs` maps not-found to `not_found`, read-only to
`failed_precondition`, insufficient space to `resource_exhausted`,
already-exists to `already_exists`, and everything else to `internal`;
the four RPCs prefix their own context with `status_with_context`, so a
message reads "commit compact volume 7: volume id 7 is not found". The
store-side "during compact" / "during commit compact" / "during cleaning
up" suffixes are gone, and the free-space message drops the volume id the
prefix already supplies.
`check_compact_volume` had no callers — `VacuumVolumeCheck` computes the
garbage level from its own `find_volume` — and is deleted. `compact_volume`
folded the size estimate into its first lookup, dropping the `unwrap()`
re-lookup that only existed to dodge a borrow.
`ascending_visit` on `CompactNeedleMap`, `RedbNeedleMap`,
`SortedFileNeedleMap` and the `NeedleMap` dispatch is now generic over the
visitor's error type, like `CompactMap::ascending_visit` already was. The
three signatures that can fail on their own bound `E: From<String>` to
carry those failures; the in-memory walk in `iter_entries` names
`Infallible`, which says in the type what its comment used to say in prose.
No Go shell command matches on the old error text: the strings exist only
in weed/storage/store_vacuum.go.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* volume server: trim comments and answer the same codes from Go
- vacuum_volume_check reports VolumeError::VolumeNotFound like the other
vacuum RPCs instead of its own "not found volume id" wording
- drop doc comments that restate what the code says
- Go volume server wraps ErrVolumeNotFound/ErrInsufficientSpace from
store_vacuum.go so VacuumVolumeCheck/Compact/Commit/Cleanup and
DeleteCollection answer NotFound/ResourceExhausted, matching the Rust
volume server; volumeDeleteStatusError generalized to volumeStatusError
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: prefix operation context on vacuum errors
Lower-level errors forwarded by CompactVolume, CommitCompactVolume,
CommitCleanupVolume and DeleteCollection carry no volume id or operation
name. Wrap with %w so the status mapping still sees the sentinel chain,
matching the context the Rust server's status_with_context adds.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: map NotEmpty to FailedPrecondition, share mapper in VolumeDelete
Go's volumeStatusError maps ErrVolumeNotEmpty to FailedPrecondition; the
Rust Status conversion was missing it and volume_delete kept a hand-rolled
match. Route it through status_with_context like the vacuum handlers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Volume carried `pub has_remote_file: bool` next to `pub volume_info`,
and the bool was only ever the answer to `!volume_info.files.is_empty()`:
outside the two constructors, `refresh_remote_write_mode` was the single
writer. Both fields being public made the pair a convention rather than
an invariant. Every caller that touched `volume_info.files` — load_vif
twice, the tier-up handler, the tier-down handler and its rollback — had
to remember to call `refresh_remote_write_mode` afterwards, and a caller
that forgot would leave the volume advertising a write mode its .vif
contradicts, or serving a remote .dat through a writable needle map.
The bool becomes `has_remote_file()`, computed from the list, so it
cannot drift. `volume_info` becomes private with a `volume_info()`
reader, and edits to the reference list go through
`update_remote_files(|files| ...)`, which applies the closure and then
refreshes the derived write mode and the needle map. With no caller left
outside the module, `refresh_remote_write_mode` is private.
Unchanged: the refresh logic itself, the order of operations in both
tier handlers, and the tier-down rollback semantics. The rollback still
snapshots the removed reference before the refresh runs, restores it on
failure, and re-refreshes unconditionally on the error path — the second
`update_remote_files` call runs with a no-op closure when there was
nothing to restore, exactly as the old code re-ran the refresh whether
or not it had re-inserted a reference.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Three production `unsafe` blocks carried no `// SAFETY:` comment at all
(`libc::fallocate`, `libc::sysinfo`, `libc::statvfs`), and nothing made
that an error: `clippy::undocumented_unsafe_blocks` is a `restriction`
lint, allow-by-default, and appeared nowhere in either crate. Turn it on
in `seaweed-volume`'s `[lints.clippy]` and in the worker workspace's
`[workspace.lints.clippy]`, then document what each block relies on.
`memory_status.rs` and `disk_location.rs` get their blocks narrowed to
the `zeroed()` and the libc call, so each comment sits next to the
operation it justifies and the arithmetic is outside the block. Both
turn the success test into an early return on failure; the casts, the
multiplication order and the values returned on either path are
unchanged.
The bigger problem was in `config.rs`'s tests. `with_temp_env_var` and
`with_cleared_security_env` called `std::env::set_var`/`remove_var`,
claiming soundness because every caller holds `process_state_lock()`.
That mutex only serialises the fourteen annotated tests in this module.
The same lib test binary runs the `grpc_server.rs` tests, which bind a
`TcpListener`, dial loopback and drive a multi-thread tokio runtime, and
tonic/hyper/rustls/aws-sdk all read the environment lazily on those
threads — which is exactly the race Rust 2024 made these calls unsafe
for. `restore_env_var` had no SAFETY comment at all. `#[serial]` would
not have helped: it serialises annotated tests, which the mutex already
did.
So the config layer no longer reads the environment implicitly. An
`EnvLookup<'a> = &'a dyn Fn(&str) -> Option<OsString>` is threaded from
the public entry points down to every reader — `HOME`, `USERPROFILE`,
the twenty-four `WEED_*` keys and `SEAWEED_WRITE_QUEUE`. `parse_cli` and
`parse_security_config` keep their signatures and pass `process_env`, a
thin wrapper over `std::env::var_os`; `resolve_config` becomes
`resolve_config_with_env` (private, one caller). Tests build one with
`fake_env` instead, so no test touches the real environment and every
`unsafe` in the module is gone.
`process_state_lock()` stays, with a smaller job: `set_current_dir` is
safe but still process-global, so the tests that move the working
directory are still serialised against the ones that read it. Tests
naming an explicit config file never reach that search and no longer
take the lock.
No production behaviour changes: the same keys are read in the same
order with the same precedence, and `env_string` reproduces
`std::env::var(key).ok()` — absent and non-UTF-8 both read as unset.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
- README and values.yaml now describe the allowInsecureBind escape
hatch alongside the non-loopback bind guard
- remove a PR reference from the CI test comment
PR #11185 made `weed admin` refuse to bind a non-loopback address
without -adminPassword or mTLS. PR #11228 added -allowInsecureNoAuth
as an explicit opt-out for operators who restrict admin access some
other way (e.g. a NetworkPolicy plus an authenticating reverse proxy).
The chart's render-time guard added by #11236 (admin-statefulset.yaml,
seaweedfs.admin.authEnabled) predates -allowInsecureNoAuth and only
recognizes password-based auth, so there was no values.yaml path to
express that choice: the chart would fail(...) even though the binary
itself would start fine with a warning.
Add admin.allowInsecureBind (default false) to the seaweedfs.admin.authEnabled
helper's checks; when true it renders -allowInsecureNoAuth on the admin
command and satisfies the render guard alongside the existing
password-based checks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Presigned HeadObject/GetObject requests hoist x-amz-checksum-mode into the
signed query string, so a strict header-only check would withhold stored
checksums on presigned reads that AWS honors.
Presigned CompleteMultipartUpload requests hoist x-amz-checksum-type and
the full-object checksum header into the signed query string, so a
header-only lookup would skip BadDigest validation for them.
COMPOSITE uploads must still carry every part checksum in the complete
request, but FULL_OBJECT uploads may instead supply the whole-object
checksum in an x-amz-checksum-* request header. Compare that header
against the computed object checksum and return BadDigest on mismatch,
matching AWS.
An UploadPart that explicitly selects a different checksum algorithm than
the one declared at CreateMultipartUpload would store a checksum
CompleteMultipartUpload could never accept. Reject the conflict up front
with InvalidRequest, matching AWS.
Parse the Checksum* elements of each completed part and enforce what AWS
does for uploads created with x-amz-checksum-algorithm: every part must
carry a checksum in the complete request (InvalidRequest when missing,
BadDigest when it differs from the stored part checksum), and an
x-amz-checksum-type header must match the upload resolved checksum type
(BadDigest). Add the issue-11401 reproduction as a regression test.
AWS computes a checksum for every part of an upload created with
x-amz-checksum-algorithm, even when the part request carries no checksum
headers. Mirror that: when the part request specifies no algorithm, apply
the one stored on the upload entry so the part entry keeps a checksum
CompleteMultipartUpload can fold into the object checksum.
fix(volume): stop scans at a header they cannot advance past
A corrupt .dat header with a very negative size gives a record length
(NeedleHeaderSize + NeedleBodyLength) of zero or less: v3 sizes -43..-36
and v2 sizes -35..-28 give exactly zero, and smaller sizes give a
negative length. ScanVolumeFileFrom advanced by that length, so it
re-read the same header forever or stepped back into the record before
it. weed fix, weed export, weed compact, incremental weed backup and the
tail sender behind volume.move and volume.merge could hang on such a
volume, and weed compact could also finish with a .cpx that had dropped
every needle after the header.
Return an error wrapping needle.ErrorCorrupted instead. The check runs
after the visitor has seen the record, so the rebuild scanner still
stops quietly with io.EOF. Smaller negative sizes whose record length is
positive are still stepped over, preserving the salvage behavior
compaction relies on.
Mirror the guard into the Rust volume scans: DatScanPlan::scan and
read_all_needles fail on a non-positive record length, as does
scan_dat_head, so a corrupt header cannot stall a tail pass or leave the
repair scan walking stale offsets.
* fix(volume): reject negative sizes in ReadNeedleBlob and WriteNeedleBlob
A ReadNeedleBlob RPC with a size of -44 or below (-36 on v2 volumes)
panics in makeslice inside needle.ReadNeedleBlob. The volume gRPC server
has no recovery interceptor, so one request kills the process. Smaller
negative sizes return bytes that are not a record.
WriteNeedleBlob accepted a negative size whenever the blob header
carried the same value: it appended the blob to .dat and indexed the
needle with that size, which reads as deleted.
Reject size < 0 in both Volume methods. Size 0 still passes, since
delete records carry it. The Rust volume server got the same storage
guards in #11345.
* fix(volume): reject needle blobs whose length does not match their size
WriteNeedleBlob appends the blob as is. A blob that is not the length
its size implies leaves .dat off the 8-byte grid, and every later
ordinary write to the volume is indexed at a truncated offset and reads
back as EOF. A blob off by 8 bytes keeps the grid but leaves bytes that
a .dat scan reads as the next record.
The in-tree callers already send exact lengths. The one case this newly
refuses is a copy between volumes of different needle versions, and
that case already writes a broken record: a v3 record lands on a v2
volume with 8 extra bytes, and a v2 record on a v3 volume either fails
the timestamp check or lands 8 bytes short.
This is separate from the negative-size guards, whose Rust counterpart
is #11345. The Rust server does not check the length yet.
* fix(volume): guard the blob buffer allocation in needle.ReadNeedleBlob
Volume.ReadNeedleBlob rejected negative sizes, but needle.ReadNeedleBlob
still sized its buffer from the size and is called directly by vacuum and
other paths. Reject a deletion marker before make() there too, and use
size.IsDeleted() in the volume-level checks.
* fix(volume): mirror the blob length check in the rust volume server
write_needle_blob_and_index checked the size against the blob header but
appended the blob verbatim, so a blob that is not the length its size
implies still leaves .dat off the record grid. Match the Go check.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(volume): return an error instead of 201 when a write lands on no volume
ReplicatedWrite only writes locally when this server holds the volume.
For a volume id no server holds, the master lookup returns no locations,
so the write went nowhere and the upload still got 201 Created. The same
happened for a type=replicate write to a server without the volume, so
the primary, or the S3 chunk fan-out, counted a replica that was never
written.
A server without the volume still forwards the write to the replicas the
master lists. When there is nothing to forward to, fail with "volume N
not found on host:port". PostHandler returns that as 500, the status the
Rust volume server already returns here, and uploaders re-assign on 5xx.
Fixes#6609
* volume: reuse Store.HasVolume, drop issue ref from test comment
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix 11388
* fix(volume): scrub validates local deletion tombstones
TombstoneFileSize (-1) is an .idx-only sentinel; the physical record it
points at carries a zero-sized body. Normalize deleted index sizes to 0
via onDiskSize before computing disk usage and calling ReadData, so
corrupted or truncated tombstone records are detected instead of
skipped. Offset-zero entries (remote logical deletes, no .dat record)
remain skipped, and the physical needle id is checked against the index
key. Mirror the behavior in the Rust volume server.
* fix(volume): scrub preserves physical size of deleted non-tombstone entries
Size.Raw()/raw() already encodes the index-to-disk mapping: tombstone
(-1) -> 0, other negative sizes -> their absolute value (the offset then
points at the original record, per the ReadDeleted path). Use it instead
of mapping every deleted size to 0.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(volume): derive needle body tail bound from the version layout
The size guard in ReadNeedleBodyBytes computed the tail length as
checksum, plus timestamp only for Version3. Forks and future on-disk
formats whose tail carries more fields would silently under-check and
still panic in readNeedleTail on a truncated body. Derive the tail from
NeedleBodyLength minus data and padding so the bound stays exact for
every version.
Iterate IsSupportedVersion in the new tests instead of hardcoding
v1-v3 so downstream formats get covered automatically, and skip
versions the build cannot write rather than failing on them.
* test: skip needle write only on the unsupported-version error
A blanket skip would hide a real writer regression. Skip the version
subtest only when the writer reports the version is not supported in
this build (the error text differs between builds), and fail on any
other write error.
* s3api: add Snowflake s3compat API integration tests
Run the upstream snowflakedb/snowflake-s3compat-api-test-suite against a
local SeaweedFS server in CI. test/s3/snowflake/run.sh starts weed server
with S3 (-s3.autoCreateBucket=false so missing-bucket PUTs return
NoSuchBucket), prepares the fixtures the suite needs (versioned bucket,
deny-all-policy bucket, >1000-object prefix), clones the suite, patches
it to path-style addressing, and runs mvn -Dtest=S3CompatApiTest.
The suite also exposed that GetBucketLocation returned 404 NoSuchBucket
for a malformed bucket name; validate the name first and return
400 InvalidBucketName like AWS.
* test: harden snowflake s3compat runner per review
- Pin the upstream suite to a tested commit (SUITE_REV) instead of the
moving default branch
- Bind the test server to loopback only
- Require the AccessDenied error code when verifying the denied bucket
- Fix README so go install runs in a subshell
- checkout with persist-credentials: false
- Make the concurrency group unique per PR, and widen path filters to
the storage/operation/wdclient/cluster/pb packages the S3 stack uses
* test: advertise loopback ip for snowflake test server
-ip.bind 127.0.0.1 alone left the volume server advertising the host's
primary address, so chunk uploads were refused. Also set -ip 127.0.0.1
and disable the Iceberg/Lance listeners so the harness is loopback-only
and does not collide with other local services.
ReadNeedleBodyBytes sliced the needle body with the size from the needle
header without checking it. A corrupted .dat header carrying size -1 still
gets a positive body length (16 bytes on v3), so vacuum compaction read
that body and panicked with "slice bounds out of range [:-1]".
Writers never put a negative size in a .dat header: a delete appends a
size-0 record, and TombstoneFileSize only lives in the .idx. Reject a size
that is negative or leaves no room for the checksum/timestamp tail with an
error wrapping ErrorCorrupted. ScanVolumeFileFrom already logs body read
errors and moves on, so compaction now skips the record like any other
corrupt needle.
Fixes#6763
* pb: stop exiting the process on malformed server addresses
ServerToGrpcAddress and GrpcAddressToServerAddress called glog.Fatalf
when hostAndPort could not parse the port, which os.Exit(255)ed the whole
process. A caller-supplied copy or tail source address reached this path
synchronously in the serving goroutine, so one anonymous VolumeCopy with
a non-numeric port terminated the volume server.
Log the parse error and return the input unchanged instead: the dial or
request that consumes the address then fails as an ordinary error.
* volume: validate copy and tail source addresses before dialing
VolumeCopy, VolumeEcShardsCopy and VolumeTailReceiver dial a
caller-supplied source address (SourceDataNode / SourceVolumeServer)
with no endpoint validation, so an anonymous caller could aim the volume
server at loopback, link-local (cloud metadata) or other unintended
destinations and read dial behavior back as a connectivity oracle.
Apply the same peer-target deny list FetchAndWriteNeedle uses for
replica targets: the source must be a bare host:port whose host is not
loopback, link-local or unspecified; cluster peers stay reachable on
private networks, and -volume.allowUntrustedRemoteEndpoints opts out.
The loopback-using copy tests set the flag to keep exercising the copy
path in process.
* rust volume: validate copy and tail source addresses before dialing
Mirror the Go guard on the Rust volume server: volume_copy,
volume_ec_shards_copy and volume_tail_receiver dial a caller-supplied
source address, so run it through validate_replica_target first (bare
host:port; no loopback, link-local or unspecified hosts; private peers
stay allowed). --volume.allowUntrustedRemoteEndpoints opts out; the test
fixture and the Rust test-cluster launcher set it so loopback sources in
tests keep working.
* volume: pin validated copy/tail source addresses at dial time
validateReplicaTarget resolves the source hostname once, but the gRPC
client resolved it again at connect, leaving a DNS-rebinding window for
hostname sources. The copy and tail source dials now run through the
same guardedDialerPolicy the remote-storage path uses, so every resolved
address is re-checked against the replica deny list (private peers
allowed) immediately before the TCP connect. guardedDialerPolicy also
moves to util.OutboundDialContext so the guarded path keeps the -ip.bind
source binding the default gRPC dialer had.
The Rust volume server mirrors this with connect_guarded, a tonic
connector that resolves, re-checks each address, and connects to the
first passing IP; handlers use it whenever the untrusted-endpoint
opt-out is off. A handler-level test now exercises the enabled
validation branches for all three source-taking RPCs.
* pb: return empty server address for malformed grpc addresses
GrpcAddressToServerAddress used to return the unparseable input on a
hostAndPort failure, so a malformed raft address (e.g. "host:abc")
flowed into admin dashboard master maps unchanged. Return an empty
string instead, skip empty conversions at the two raft-cluster merge
sites, and drop the now-stale comment about the fatal exit the earlier
commit removed.
* test: opt erasure-coding loopback clusters out of the remote endpoint guard
The erasure-coding suites drive VolumeEcShardsCopy / VolumeCopy between
volume servers bound to 127.0.0.1, which the copy/tail source guard now
rejects by default. Pass -volume.allowUntrustedRemoteEndpoints to the
test volume launches, matching what the volume_server framework
harnesses already do.
* admin: only claim fallback master leadership on an empty raft response
A nonempty RaftListClusterServers response whose entries were all
rejected left masterMap empty, so the fallback marked the reachable
current master as leader the same way a genuinely empty (non-raft)
response does. Track whether the successful response returned zero
servers and only promote the fallback master then.
* s3api: test for chunks deleted under an entry the filer committed
Issue #11387: the filer can report a create failure after inserting the
entry (e.g. a parent-directory creation failing post-insert). The error
arrives in the response rather than as a transport status, so it maps
to a definitive error and putToFiler deletes the chunks of the live
entry.
* s3api: confirmCreateLanded also reports a confirmed-absent entry
The verification a failed create runs can answer both directions: the
entry matching the uploaded chunks proves the write landed, and an
authoritative not-found proves the uploaded chunks are orphaned. Return
both outcomes so the cleanup path can gate on the fact rather than the
error class. An empty upload can never prove a landing, so a zero-chunk
entry match no longer upgrades the outcome.
* s3api: delete orphaned chunks only when the entry is confirmed absent
A failed create no longer skips verification based on the error class: the filer can fail after inserting the entry (issue #11387) and a partially-applied routed transaction can leave it behind too, both surfacing as definitive errors. Every failed create now resolves the entry's fate, and the uploaded chunks are deleted only when the entry is confirmed absent; anything unverifiable keeps them for vacuum.
* s3api: confirm absence on every filer the create could have committed on
A lock-path create fails over across filers, so the entry can live on a replica the routed owner has not caught up to; one not-found does not prove absence. The confirmation now queries the owner, the prior owner, and the failover set, declaring absent only when none of them has the entry.
* s3api: bound the reconciliation lookups confirmCreateLanded runs
The lookups ran on context.Background() under the object write lock, so a connected filer that never replies could stall the write path. One timeout now covers the whole enumeration; an expired budget fails the remaining lookups as uncertain, which keeps the chunks.
* fix(filer): leave reader cache unbounded without an explicit budget
NewReaderCache silently installed a 256MiB ReaderCacheBudget when the
caller passed none. Only weed mount opts into a budget; every other
caller (S3 gateway, WebDAV, query engine, mq logstore) inherited the
cap. Under ~90 concurrent S3 GETs of medium objects, prefetch wants
far more than 64 chunk buffers, so reserve() serialized chunk fetches,
clients timed out and retried, and the retry re-downloaded chunks the
cancelled request had already fetched.
A nil budget now means unbounded, restoring the pre-4.47 behavior for
callers that never asked for a memory cap; reserve/complete/release
are nil-safe. The mount path is unchanged and still enforces
-readerCacheSizeMB.
Fixes#11380
* feat(s3): expose -s3.readerCacheSizeMB reader buffer budget
Operators who want the S3 gateway read path memory-bounded can now
opt in: -s3.readerCacheSizeMB on weed filer/server/mini and
-readerCacheSizeMB on standalone weed s3, matching the mount flag.
The default 0 keeps the unbounded pre-4.47 behavior; a positive value
installs a shared ReaderCacheBudget across in-flight and retained
chunk buffers for all S3 GETs.
* fix(filer): validate chunk size before consulting the reader budget
A nil budget returned early and skipped the negative chunkSize check,
letting a corrupted size reach mem.Allocate and panic. Also drop the
command-specific flag prefix from the S3 validation error since
standalone weed s3 exposes the option as -readerCacheSizeMB.
* filer: drop chunk buffers once fully consumed
ReaderCache retained every completed chunk buffer in the downloaders
map until the slot limit evicted it, so buffers lingered after all
readers finished with them.
Track attached readers on each SingleChunkCacher and remove the cacher
when the last reader consumes the buffer to its end. In-flight download
deduplication and the prefetch handoff are unchanged: a buffer always
survives until fully read, partial reads keep it available, and an
attached reader pins a consumed buffer until it detaches. Repeat reads
now go through the chunk cache where enabled, or refetch.
* filer: drop consumed buffers on last detach, rechecked under cache lock
Two review findings on the drop-on-consume change:
- Removal only fired when the detaching reader itself reached the chunk
end. If the end-reaching reader finished first and the last remaining
reader did a partial read or cancelled, the consumed buffer and its
budget reservation lingered until eviction. Track a persistent
consumed flag instead, so any end-reaching read marks the buffer and
the last detach drops it.
- remove() checked only map identity, so a reader attaching between the
reader count hitting zero and removal could attach to a cacher that
was then deleted underneath it. removeConsumed() re-checks identity,
readers == 0, and consumed under the ReaderCache lock; a raced attach
keeps the cacher and its own detach retries the removal.
* s3: abort a completed upload's leftover directory metadata-only
A .uploads/<id> directory can outlive the object it completed into when
the commit's metadata-only removal failed or the gateway died in between;
the restored part entries then share chunks with the published object.
AbortMultipartUpload deleted the directory recursively, chunks and all,
so aborting such a leftover destroyed a committed object (#11382).
Run the same check s3.clean.uploads gained in #11375 before deleting:
when the object entry or a version file under <key>.versions carries the
upload id, remove .uploads/<id> metadata-only and answer the abort; when
the lookup cannot decide, refuse with InternalError rather than risk
live chunks.
* s3: apply the completed-upload check to lifecycle MPU abort
lifecycleAbortMPU ran the same destructive recursive delete on
.uploads/<id>. Reuse uploadCompleted so a leftover whose object entry or
version file carries the upload id is removed metadata-only, and an
undecidable lookup retries later instead of freeing live chunks.
* s3: serialize abort's upload-dir delete with the object's commit
The completed check alone leaves a race: abort can read completed=false,
then an in-flight completion publishes the object over the same part
chunks before the recursive delete frees them.
Run the check and delete inside the object write lock, which non-routed
completions hold for their whole finalize. With an owner, send the data
delete as an ObjectTransaction on the object's lock key — a routed
commit then either loses its upload-exists precondition after our delete
or has already stamped the object, which the transaction's
IF_EXTENDED_NOT_EQUAL condition detects and falls back to a
metadata-only remove. lifecycleAbortMPU shares removeUploadDir so both
callers get the same ordering.
* s3: check for an empty object before resolving its write owner
* s3: check completion at the abort's resolved object key
An upload record missing ExtMultipartObjectKey skipped the completed
check entirely even though the request's Key names the object.
* s3api: fail closed when S3 Tables signature verification fails
* s3api: avoid nil Account dereference in S3 Tables auth log
* iceberg: return auth error instead of falling back to DefaultAllow
* lance: return auth error instead of falling back to DefaultAllow
* s3api: stop trusting client-supplied s3-account-id
The header is set by the server after successful authentication; scrub
inbound values alongside the other internal headers, and apply the same
admin guard to the header fallback branch of getAccountID that the
identity branch already has.
* test: cover table-catalog auth wrappers and principal resolution
* test: configure anonymous identity where catalog clients do not sign
* s3api: scrub s3-account-id after signature verification
* s3api: test that a session token must not reveal its credential
* sts: derive secret access key with HMAC keyed on the signing key
* s3api: stop accepting STS session tokens as bearer credentials
* security: reject STS session tokens on filer and admin gRPC auth
* test: sign s3/iam framework requests with the session credential
* s3api: exercise the real auth pipeline in the end-to-end harness
* s3: commit versioned multipart upload in one transaction
CompleteMultipartUpload wrote the version file, flipped the .versions
pointer, then removed .uploads/<id> metadata-only as a best-effort
post-commit step. A filer error or gateway crash in that window left the
upload directory referencing the same chunks as the published object, and
the next s3.clean.uploads run purged it with data -- corrupting a
committed object.
Put the version file, remove the upload directory metadata-only (its
chunks are the object's chunks), and recompute the latest pointer in one
ObjectTransaction under the object's per-path lock on the owner filer.
The mutation order keeps every partial state safe: the chunks stay
referenced at all times, and a published object never coexists with the
upload directory the cleaner would purge.
Unused part entries are freed before the transaction, since the
metadata-only directory delete would otherwise leak their chunks.
* s3: remove upload directory inside the multipart object PUT
The same committed-object/stranded-upload window existed on the
suspended and non-versioned paths: writeMultipartObject committed the
object, then a best-effort rm dropped .uploads/<id>. Ride the
metadata-only removal on the routed PUT itself so the two land in one
transaction; the unrouted mkFile fallback keeps post-commit cleanup.
* shell: purge completed uploads metadata-only in s3.clean.uploads
A leftover .uploads/<id> can outlive a committed object when the
completion's metadata-only delete fails or the gateway dies in between;
its part entries then share chunks with the live object, and a recursive
purge frees them out from under it.
Before purging a stale upload, check whether it completed: the object
entry or any version file under <key>.versions carrying the upload id.
If so, delete with skipChunkDeletion. If the lookup fails, skip the
upload for this run rather than risk live chunks.
* s3: abort multipart completion when unused part cleanup fails
Deleting the upload directory metadata-only erases the only metadata
pointing at part entries whose deletion failed, orphaning their chunks.
Propagate the error so the completion fails while the upload directory
still exists and the request remains retriable.
* s3: require the upload directory to exist at multipart commit
A delete that does not take the object lock (abort, lifecycle,
s3.clean.uploads) can remove .uploads/<id> and its chunks between the
prepare step and the commit transaction. The commit now carries an
IF_EXISTS precondition on the upload directory so the race fails the
request with NoSuchUpload instead of publishing an object over freed
chunks.
* s3: keep the version file when the upload directory is gone
The finalize transaction has no rollback, so a failure at the
latest-pointer recompute leaves the version written and .uploads/<id>
removed. Deleting the version then destroys the only remaining record of
the upload, making a retried CompleteMultipartUpload return NoSuchUpload
while the version's chunks leak. Roll back only while the upload
directory survives; otherwise keep the version, which a retry resolves
through SeaweedFSUploadId and the version reconciler promotes.
* s3: keep manifests when a routed object write partially commits
For non-versioned and suspended completions the object PUT precedes the
upload-directory DELETE, so an error can mean the object entry exists
while the response reports failure. Freeing this attempt's manifest
chunks then destroys the committed object. Keep them when the object
entry survived, and after a failed null-marker finalize which always
follows a committed write.
* s3: skip the keep-version path on precondition failure
A rejected precondition means no mutation ran, so there is no version
file to preserve and this attempt's manifests are orphans the error
cleanup should free.
* s3: keep manifests when the object-existence check itself fails
A transient lookup error previously read as absent, letting the error
cleanup free manifest chunks a committed object still references.
* s3: keep the upload directory when post-commit part cleanup fails
Removing it metadata-only after a failed entry delete erases the only
reference to the leftover chunks. Leave the directory so the entries
keep their chunk references for s3.clean.uploads or manual recovery.
* pb: fix filer list entry counting on 32-bit
int(limit) wraps to -1 on 386 when limit is math.MaxUint32, so the
beyond-limit check discarded every streamed entry. Compare in uint64
instead; the semantics are unchanged on 64-bit platforms.
* shell: resolve trailing-slash object keys in s3.clean.uploads
Completion stores a key ending in / inside the directory it names
(<bucket>/dir/dir), but FullPath+DirAndName on the normalized key
looked one level too high. Deriving dir and name with path.Dir and
path.Base mirrors getEntryNameAndDir so the completed-upload check
finds the entry instead of purging its chunks.
* s3: heal a suspended completion hidden behind a delete marker
Removing .uploads/<id> inside the commit transaction means a failed
finalizeSuspendedNullWrite leaves nothing to retry against: the object
entry is committed but the marker still makes the key read as deleted,
and a retried CompleteMultipartUpload can only report NoSuchUpload.
When the upload directory is gone, check the regular path for an entry
carrying the upload id and re-run the marker finalize, so the retry
both succeeds and repairs the key. Only suspended buckets can hold
this state; anything newer owns the key.
* s3: report store errors when resuming a committed multipart object
* mount: keep a deliberate bucket allow-empty-folders setting
* shell: s3.bucket.allowEmptyFolders toggles the empty folder cleaner
* shell: guard allow-empty-folders toggle with expected extended attrs
* filer: drop cached empty-folder policy on bucket entry update
* mount: guard allow-empty-folders write with expected extended attrs
* filer: skip caching a stale cleanup policy read across an update
* mount, shell: snapshot the full extended attributes for update preconditions
* filer: fail closed and invalidate on all bucket entry events for cleanup policy
* filer: key the cleanup policy generation by bucket
* filer: requeue cleanup when the bucket policy cannot be loaded
* filer: expire idle cleanup policy generations
* filer: skip requeueing cleanup after the cleaner stops
* filer: bound cleanup retries on repeated policy failures
* filer: cover cleanup requeue on repeated policy failures
* filer: keep cleanup policy generations while reads are in flight
* filer: exercise the cleanup queue lifecycle in the retry-cap test
* telemetry/server: tidy module dependencies
* ci: make telemetry deploy work on Oracle Linux 7
* ci: install telemetry unit and logrotate on every deploy
* ci: abort telemetry deploy on install failures
* s3api: map ambiguous filer transport errors to retryable 503
Canceled, DeadlineExceeded and Unavailable can be returned after the
filer applied the write, so the outcome is ambiguous. Reporting them as
a 4xx tells the client not to retry; report ServiceUnavailable instead.
* s3api: verify entry existence before deleting orphaned chunks
A failed CreateEntry can still have landed on the filer when the error
is a transport failure, and entryCreated=false would tombstone chunks a
live entry references, leaving a dangling pointer that survives only
because reads pass readDeleted=true until vacuum reclaims the needle.
Before deleting, look the entry up: if it is stored with the same
chunks, the write succeeded; if the lookup cannot be answered, keep the
chunks for vacuum to reclaim; only a confirmed absence still cleans up.
* s3api: regression tests for ambiguous CreateEntry outcomes
Covers the three post-create-failure cases in putToFiler: the entry
landed despite the error (treat as success, keep chunks), the entry is
confirmed absent (delete orphans), and the outcome is unverifiable
(keep chunks, return error).
* volume: count reads served from deleted needles
A readDeleted read succeeding on a tombstoned needle is the signal that
metadata still points at deleted data. Count it under a
readDeletedNeedle handler label in both the Go and Rust volume servers
so the condition is visible before vacuum turns it into a 404.
* s3api: never delete chunks on an ambiguous create error
Review feedback on the first fix showed verification could still go
wrong in both directions: a stale or lagged lookup could report
not-found for a committed entry, a prefix object stores its chunks on a
directory entry, and filer-side manifestization rewrites the top-level
chunk ids the comparison relied on.
Rework the rule so the outcome classes are asymmetric:
- A transport-level error (anything filerErrorToS3Error maps to a
retryable 503) is ambiguous and never deletes chunks; the lookup can
only upgrade the write to success.
- Any other error is a definitive filer refusal and still cleans up.
confirmCreateLanded asks the write owner first, resolves the stored
entry through chunk manifests, requires an exact match of the uploaded
file ids, and on success runs the finalize callback the failed create
skipped (under the object write lock, with the same rmObject undo the
create path uses). Zero-chunk writes stay ambiguous since they cannot
be told apart by chunks.
* s3api: cover definitive refusals and stale entries in put tests
The confirmed-failure case now uses a definitive refusal so it still
exercises orphan cleanup, and a new case keeps chunks when the stored
entry belongs to an older object rather than this PUT.
* volume: count deleted-needle reads once per request
Streamed Go reads ran the deleted check in readNeedle and again in
readNeedleDataInto, and non-streamed Rust reads in stream_info and the
full-read fallback, double-counting one request. Count at the single
entry probe each implementation takes per GET: readNeedle in Go,
read_needle_stream_info in Rust.
* s3api: run recovered-write rollback under the object lock
Two follow-ups from review: ResolveChunkManifest returns traversed
manifest blobs in its manifestChunks output, so requiring it empty
rejected every manifestized landing; and the rmObject undo ran after
the object write lock was released, so a concurrent newer write could
be deleted between finalize failure and rollback. Compare only the
resolved data chunks and keep the undo inside the lock.
* s3api: verify, finalize and roll back recovered creates in one lock
A lookup done before the object write lock let a concurrent PUT replace
the entry between the chunk comparison and the finalize/rollback
section, so a failed afterCreate could rmObject a newer write. Run the
owner lookup, manifest resolution, chunk comparison, afterCreate and
the conditional undo inside a single withObjectWriteLock section.
* security: require go 1.26.6 and bump vulnerable deps
A dependency scan of the 4.47 release flagged the bundled toolchain
and modules:
- github.com/golang/go < 1.26.6 (CVE-2026-39821, CVE-2026-56853,
CVE-2026-56859, CVE-2026-56862, CVE-2026-56864, CVE-2026-56865,
CVE-2026-33818, CVE-2026-46600): raise the go directive to 1.26.6
so every built artifact requires the fixed toolchain.
- google.golang.org/grpc (CVE-2026-84445, CVE-2026-84304): move to
the fixed dev pseudo-version; released tags through v1.85.0-dev
remain in the affected range.
- github.com/pelletier/go-toml/v2 <= v2.4.2 (unbounded parser
recursion): v2.4.3.
- alpine libcrypto3/libssl3 < 3.5.8-r0 (CVE-2026-75803,
CVE-2026-63073, CVE-2026-63075, CVE-2026-63076, CVE-2026-63072,
CVE-2026-54874, CVE-2026-18798, CVE-2026-14456, CVE-2026-14457):
the release images already apk-upgrade the final stage; extend the
same to the telemetry and admin-integration images.
Same bumps applied to the test/kafka, test/sftp,
kafka-client-loadtest, and telemetry/server modules.
* telemetry: send integration test report above the 10 GiB floor
The collect endpoint keeps reports only when TotalDiskBytes >=
proto.MinDiskBytes, but the integration test still sent 1 GiB, so the
server counted the report and skipped storing it. No cluster_id series
was ever created and /metrics lacked seaweedfs_telemetry_volume_servers.
Send just above the floor (via proto.MinDiskBytes so it cannot silently
drift again) so the expected per-cluster metrics are exported.
* rust volume: one positional read helper; never seek a dup'd handle on Windows
Positional read-exact was hand-rolled four times: the complete
cross-platform version in needle_map/sorted_file.rs, a Windows-only half
in volume.rs whose unix half was inlined as a
cfg(unix)/cfg(windows)/compile_error! triple at three call sites, a
byte-identical Windows-only copy in ec_volume.rs, and read_full_at in
ec_bitrot.rs. Three more sites -- EcVolumeShard::read_at,
EcLocalShard::read_at and ec_encoder::read_at_most -- hand-rolled the
short-read-permitted variant with a cfg(not(unix)) arm that
try_clone()s the handle and seeks it.
That last arm is wrong. A duplicated descriptor shares one kernel file
offset with the original, so seek-then-read is two syscalls against
state another thread can move in between: a concurrent reader or an
append repositions the offset and the read returns bytes from somewhere
else entirely. EcLocalShard::read_at documents that it must never seek,
one line above the seek. Windows seek_read carries its own offset in a
single call, so that window does not exist.
All seven now go through storage::io::{read_exact_at, read_at}, whose
module doc records why duplicating a handle is not a way to get a
private file position -- opening the file again is, as
Volume::dat_scan_plan already does. read_at_most keeps its own
fill-until-EOF loop; only the per-iteration positional read changes.
Behaviour on unix is unchanged: every unix arm was already
FileExt::read_exact_at or FileExt::read_at. The one exception is
ec_bitrot::verify_shard_blocks, which now retries on EINTR (std's
read_exact_at does; the loop it replaces did not) and, on unix, reports
the standard "failed to fill whole buffer" text instead of "short read
on shard block". The Windows arm still says "unexpected EOF in
seek_read"; both carry ErrorKind::UnexpectedEof, as before.
NeedleStreamSource::read_exact_at and Volume::read_exact_at_backend keep
their signatures; only their bodies shrink.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: retry Interrupted in Windows read_exact_at
Unix std's FileExt::read_exact_at ignores ErrorKind::Interrupted and
retries, but the Windows seek_read loop propagated it, so the shared
exact-read contract differed by platform. seek_read can surface
ERROR_OPERATION_ABORTED, which std maps to Interrupted.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* s3tables: hide inaccessible table buckets
* s3tables: hide inaccessible namespaces
* s3tables: hide inaccessible tables
* s3tables: hide inaccessible resources in rename and namespace delete
RenameTable/RenameView denied on the source now report the same
not-found as a missing source, and the destination name conflict is
checked only after destination authorization so a denied caller cannot
distinguish an existing destination namespace or name from a missing
one. DeleteNamespace denials use the same formatted message as a
missing namespace.
* fix(ec): BatchDelete cookie fail-closed via locate_data geometry
* fix(ec): honor skip_cookie_check, require full cookie header
* fix(ec): retry short cookie header reads, still fail closed on EOF
* chore(ec): trim cookie validation comments
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Serve can return when its listener closes while HTTP requests are still draining. The main path could then close the metadata store before those requests finish.
Make signal, context, and Serve-exit paths join one shutdown sequence. Drain gRPC and HTTP concurrently with 15-second default limits, then close the store. Test both completion orders.
fix: flush metadata log before closing filer store
Serialize sealed-batch handoffs with shutdown, reject late appends, and wait for log-buffer workers before closing the filer metadata store.
Cover queued writes, interval and explicit flushes, late-write rejection, and pending persistence with shutdown tests.
handlers.rs split needle URLs in three places and the three disagreed.
Go does it once, in parseURLPath (weed/server/common.go:218-249), and
dispatches on the slash count: /vid/fid/filename takes the extension off
the filename and leaves the fid whole, /vid/fid takes it off the fid, and
the comma form splits the last segment on its last comma and dot.
Two of the Rust copies got that wrong:
- extract_file_id returned the path unchanged when it found no comma, so
a JWT fid claim, which Go compares against vid + "," + fid for every URL
form (volume_server_handlers.go:361-364), could never match a slash-form
request. With a JWT key configured, every read, write or delete of
/3/01637037d6 was a 401.
- build_proxy_request_info's slash branch had no extension handling, so a
redirect for /3/01637037d6.jpg sent the client to /3,01637037d6.jpg.
Go's proxyReqToTargetServer formats "%s/%s,%s" from the already-stripped
fid (volume_server_handlers_read.go:128-137) and so emits
/3,01637037d6. The peer still serves either form, since the comma form
strips the extension again, so this one is parity rather than breakage.
Replace all three with one parse_needle_path returning vid, fid, ext and
filename borrowed from the path. The fid keeps its _delta suffix, as in
Go: parse_needle_id_cookie applies it and the JWT check strips it. The
leading slash stays optional, so chunk manifest fids still parse.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* shell: fs.verify -pruneEntries deletes entries whose needles are lost
* shell: harden fs.verify -pruneEntries guards; VolumeNeedleStatus returns NotFound for absent needles
* shell: resolve chunk manifests in fs.verify metadata path; require confirmed deletion before counting prunes
* shell: anchor fs.verify legacy missing-needle error matching
* shell: keep fs.verify metadata scan alive on manifest resolution failures
* shell: classify EC missing needles and keep manifest failures unverified
VolumeNeedleStatus now canonicalizes erasure_coding.NotFoundError to
codes.NotFound, so absent needles in EC volumes reach the prune path
through the same stable contract as regular volumes. The client-side
isNeedleMissingError keeps recognizing the legacy wrapped EC shape
("locate in local ec volume: ... needle not found") for mixed-version
clusters.
A chunk manifest that fails to resolve is now an entry-level
verification failure even when the raw top-level chunks are healthy:
the file is not fully readable without the manifest. Raw chunks are
still verified on a resolution failure so a missing top-level manifest
needle is classified and can be pruned. The per-entry logic is
extracted into resolveAndVerify for testability.
* shell: trim fs.verify prune comments
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: a list marker that sorts before the prefix excludes nothing
ListObjects `marker` and ListObjectsV2 `start-after` are a plain key
cutoff: list the keys that sort after it. A marker that sorts before the
prefix and is not under it therefore excludes no key carrying the prefix,
and the listing must equal the one with no marker at all.
normalizePrefixMarker treated every marker that does not start with the
prefix as "something wrong" and the listing came back empty. Clients send
this shape routinely: docker/distribution's S3 storage driver walks
prefix "<root>/<path>/" with start-after "<root>" (its rootdirectory), so
on SeaweedFS a registry walk saw an empty bucket. zot read that as "no
repositories": /v2/_catalog was empty, GC/scrub/retention never saw a
repo, and on restart its storage parse deleted every repository's
metadata as "no longer in storage".
listFilerEntries now lists as if no marker were given when the marker
sorts before the prefix; the response still echoes the marker the client
sent. A marker that sorts after the prefix's subtree is left alone: it may
legitimately sit inside a partial-name prefix's match set, which
normalizePrefixMarker already handles, and otherwise correctly lists
nothing.
Reproduce on 4.44 and 4.47:
curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=zot/zot/" # all keys
curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=zot" # KeyCount 0
curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=a" # KeyCount 0
* s3: keep the prefix's own key excluded by a marker that names it
Fold the before-prefix marker rule into normalizePrefixMarker, which now also
derives prefixEndsOnDelimiter from the effective marker instead of each cursor
rebuilding the expression.
A marker equal to the prefix is no longer trimmed to a subtree cutoff:
start-after "a/b/" with prefix "a/b/" excludes only the "a/b/" key, so the walk
starts inside that directory and its children still list.
Adds a listing-level test that walks the whole path for both start-after shapes
a registry sends, and covers the new normalization cases.
* s3: leading slashes do not hide a marker that names the prefix
* s3: echo the V1 marker the client sent, not the walk's cutoff
* s3: filter only the walk's cutoff from the V1 page, not the echoed marker
* s3: skip the key an exclusive marker names as it streams
---------
Co-authored-by: Zuse <be9c90a8-c104-4be2-b7a4-9f92eb833ac8@forge.local>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust volume: add a .dat scan plan that runs without the store lock
DatScanPlan captures a fresh .dat handle, the version, the start offset
and an end bound while the caller holds a store guard, then visits one
record at a time with positional reads that never touch the Volume, the
way Go's ScanVolumeFileFrom feeds a scanner.
The handle pins the inode the offset was resolved against: a vacuum
commit renames .cpd over .dat and destroy unlinks it, and neither
rewrites the pinned bytes. The end bound is read while no writer can
hold store.write(), so the scan never meets a partial append. It is a
fresh open, not try_clone, because on Windows read_exact_at uses
seek_read, which moves a cursor a clone shares with the writer.
A header whose size is negative, or does not fit before the end bound,
ends the pass before the body length is computed or anything is
allocated. In today's scan a negative size reaches needle_body_length
and either overflows the buffer size or walks the scan from a wrong
offset. A size near i32::MAX overflows padding_length's i32 arithmetic,
which panics in debug builds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3
* rust volume: stream the tail scan with the store lock released
volume_tail_sender read every needle from the start offset to EOF into a
Vec while holding store.read(). volume.merge tails from zero, so that was
the whole volume in memory. And because needle writes and the heartbeat
take store.write() on a lock that prefers writers, the whole node stopped
serving until the scan finished: the failure #11235 fixed for EC scrub.
Each pass now runs on a blocking thread. Under one store guard it
resolves the start offset and captures a DatScanPlan, then drops the
guard and sends each needle as it is read, as Go's
VolumeFileScanner4Tailing does. This replaces the one-guard-across-
search-and-scan rule from the previous commit with a stronger invariant:
the offset, the handle and the end bound come from the same guard, and
the handle pins the inode, so a vacuum commit mid-scan cannot point the
offset into the compacted file.
A scan error now ends the stream with Status::internal instead of a
clean EOF, as Go's `streamFollow: %w` does. Once needles stream, a clean
EOF after a partial pass would let volume.move treat a truncated tail as
complete. A panic in the pass is reported the same way. A receiver that
hangs up is also noticed between skipped needles, not only on a send.
Unchanged: the append_at_ns filter, the header on every 2MB chunk, the
caught-up heartbeat without a scan, and the draining countdown.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3
* rust volume: fail the tail pass on a short read below the snapshot end
DatScanPlan::scan treated an UnexpectedEof on the header or body read as
the end of the data and returned Ok. Every byte below the captured end
existed when the plan was taken, so a short read there can only mean the
inode was truncated under the plan: an unmount followed by a VolumeCopy
of the same volume id reopens .dat with truncate(true). The pass then
reported Scanned, the next pass found the volume gone, and the stream
ended cleanly after a prefix of the planned records, which volume.move
would take as a complete tail.
Both short-read arms now fail the scan with an I/O error that names the
offset and the snapshot end, so tail_pass reports Status::internal as it
does for every other read failure. The break arms were carried over from
scan_raw_needles_from, where the whole scan ran under the store guard and
nothing could truncate the file.
Found by the Devin and Greptile reviews on #11275.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: sum the needle padding in i64 so a corrupt size cannot overflow
padding_length added the header, checksum and timestamp widths to the
needle size in i32. A size read from a corrupt header can sit near
i32::MAX, and that sum then overflows: a panic with overflow checks, a
wrapped padding without. DatScanPlan::scan bounds the size against the
bytes left before computing the body length, but that only keeps such a
size out of the arithmetic while under 2 GiB of the file remains, so on a
large volume the scan could still reach the overflow and, in release,
size a buffer from garbage.
Sum in i64 in both version branches. The result is at most
NEEDLE_PADDING_SIZE, so it still fits Size. The scan comment no longer
claims the bound check prevents the overflow.
Found by the CodeRabbit review on #11275.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust volume: propagate dat scan parse failures
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: roll master pods when master config changes
The master loads master.toml once at startup (startAdminScripts reads
master.maintenance.scripts and sleep_minutes via viper with no config
watching), and the master ConfigMap is mounted with subPath, which
kubelet never refreshes in a running pod. So a change to
.Values.master.config today updates the ConfigMap but running masters
keep executing the old configuration until something else restarts
them.
Add a checksum/config annotation on the master pod template, following
the existing checksum/s3config pattern on the filer and s3 pods, so a
master config change triggers a rolling restart of the masters.
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* Guard against duplicate keys
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* Add checksum to deployment as well
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* Always ensure the annotation is set
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* Update comments
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* Soften stance
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
* helm: merge pod annotations before checksums
---------
Signed-off-by: Evans Mungai <mbuevans@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust: migrate seaweed-volume and seaweed-worker to tonic 0.14 / prost 0.14
tonic 0.14 boxes the contents of tonic::Status, which is what made every
RPC path trip clippy's result_large_err; the allow for that lint goes in
the next commit. The prost codec moved out of tonic into tonic-prost and
tonic-prost-build, so both build scripts now call
tonic_prost_build::configure() and both crates depend on tonic-prost for
the generated code. The `tls` feature was split into a per-backend
feature; `tls-aws-lc` is the same backend both crates already install
through rustls::crypto::aws_lc_rs.
tonic 0.14 depends on axum 0.8 and tower 0.5, which would have left a
second axum and a second tower in each tree next to the 0.7 / 0.4 the
crates named themselves. Bumping them keeps one copy of each: axum 0.8
only changes the path-parameter syntax for the routes here (`/:vid` ->
`/{vid}`, `/*path` -> `/{*path}`), tower 0.5 needs the `util` feature
named explicitly for ServiceExt::oneshot (it used to arrive through
tonic's feature unification), and tower-http 0.6 is the matching
release.
Lock files move only through cargo's own resolution for the new
versions; no other dependency was refreshed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust: drop the result_large_err allow now that tonic::Status is boxed
tonic 0.14 stores Status behind a Box, so Result<_, Status> is no longer
a large-Err type and clippy has nothing to say about it. Both crates
pass `cargo clippy --all-targets -- -D warnings` without the allow
(seaweed-volume in both feature sets), so the policy entry and its
comment go.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: drop the unused headers argument of try_expand_chunk_manifest
The parameter was already named `_headers`; nothing in the body reads it.
With it gone the function is under clippy's argument threshold and the
expect goes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: pass EC peer reads an EcInterval instead of ten arguments
fetch_one_interval, read_remote_ec_shard_interval,
do_read_remote_ec_shard_interval and recover_one_remote_ec_shard_interval
all took the same (vid, needle_id, shard_id, shard_offset, size,
expected_encode_ts_ns) tuple, and the two that reconstruct also took the
location map with the data/parity counts. Those are now EcInterval (Copy)
and EcShardMap (a borrow of the map plus the counts). The fan-out inside
recovery builds its per-shard request with `EcInterval { shard_id: sid,
..iv }`, which is the one place the old argument list was easy to get
wrong. Bodies destructure at the top, so the code below the signatures
is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: give the EC encoder an EcEncodeLayout and an EncodeRun
encode_dat_file took the Reed-Solomon shape and three block sizes as five
loose integers; they are now one Copy struct, EcEncodeLayout, which is
what Go calls ECContext. The per-row and per-batch helpers took the same
six sinks and the offsets; they become methods on EncodeRun, which owns
the borrows for one run, so each call names only the offset and block
size that vary. The byte-level work is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: describe a .dat rebuild with DatRebuild instead of nine arguments
write_dat_file_from_shards, its _with_dirs twin and the private
write_dat_file were three layers over one nine-argument signature. One
public function now takes a DatRebuild, whose shard_dirs is None when
every shard sits beside the .dat and Some(dirs) for the cross-disk
reconciled layout. The field docs carry what the function doc used to
say about the encode-time size and the block layout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: split copy_file_from_source's fifteen arguments into two structs
CopyFileSpec is the per-file request (what to ask the source for, where
it lands, whether its bytes count as progress); CopyProgress is the
sender, throttler and report state that all three files of one
VolumeCopy share, held by &mut across the calls. The three production
call sites now read as the .dat/.idx/.vif literals they are, instead of
positional trues and falses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: create volumes from a VolumeSpec
Volume::new, DiskLocation::create_volume and Store::add_volume each
took the same five-value tail of Go's NewVolume argument list:
collection, replica placement, TTL, preallocation and needle version.
That tail is now VolumeSpec, a Copy struct whose Default is what almost
every test wanted anyway (empty collection, no replication, no TTL, no
preallocation, current version), so most of the 104 call sites shrink
to `&VolumeSpec::default()` or name the one field they set. The id,
directories, index kind and disk type stay positional because they
differ at every site.
Two imports that only test modules use moved into those modules, and
DiskLocation no longer imports ReplicaPlacement.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-worker: run cargo fmt
Layout only; no token in the workspace changes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: run cargo fmt
Layout only; no token in the crate changes. Every earlier Rust PR here
formatted only the blocks it touched so as not to drown its diff in
this one, and this commit is that debt paid in a single place. rustfmt
needed two passes to settle one block in handlers.rs; the committed
form is the fixed point, so `cargo fmt --check` is clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* ci: add a commented-out cargo fmt --check step to both Rust workflows
Same shape as the commented clippy step from #11312: the check is
written out so that making formatting a gate is a one-line uncomment,
and whether to do that stays a maintainer call.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* rust: migrate seaweed-volume and seaweed-worker to tonic 0.14 / prost 0.14
tonic 0.14 boxes the contents of tonic::Status, which is what made every
RPC path trip clippy's result_large_err; the allow for that lint goes in
the next commit. The prost codec moved out of tonic into tonic-prost and
tonic-prost-build, so both build scripts now call
tonic_prost_build::configure() and both crates depend on tonic-prost for
the generated code. The `tls` feature was split into a per-backend
feature; `tls-aws-lc` is the same backend both crates already install
through rustls::crypto::aws_lc_rs.
tonic 0.14 depends on axum 0.8 and tower 0.5, which would have left a
second axum and a second tower in each tree next to the 0.7 / 0.4 the
crates named themselves. Bumping them keeps one copy of each: axum 0.8
only changes the path-parameter syntax for the routes here (`/:vid` ->
`/{vid}`, `/*path` -> `/{*path}`), tower 0.5 needs the `util` feature
named explicitly for ServiceExt::oneshot (it used to arrive through
tonic's feature unification), and tower-http 0.6 is the matching
release.
Lock files move only through cargo's own resolution for the new
versions; no other dependency was refreshed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust: drop the result_large_err allow now that tonic::Status is boxed
tonic 0.14 stores Status behind a Box, so Result<_, Status> is no longer
a large-Err type and clippy has nothing to say about it. Both crates
pass `cargo clippy --all-targets -- -D warnings` without the allow
(seaweed-volume in both feature sets), so the policy entry and its
comment go.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: drop the unused headers argument of try_expand_chunk_manifest
The parameter was already named `_headers`; nothing in the body reads it.
With it gone the function is under clippy's argument threshold and the
expect goes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: pass EC peer reads an EcInterval instead of ten arguments
fetch_one_interval, read_remote_ec_shard_interval,
do_read_remote_ec_shard_interval and recover_one_remote_ec_shard_interval
all took the same (vid, needle_id, shard_id, shard_offset, size,
expected_encode_ts_ns) tuple, and the two that reconstruct also took the
location map with the data/parity counts. Those are now EcInterval (Copy)
and EcShardMap (a borrow of the map plus the counts). The fan-out inside
recovery builds its per-shard request with `EcInterval { shard_id: sid,
..iv }`, which is the one place the old argument list was easy to get
wrong. Bodies destructure at the top, so the code below the signatures
is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: give the EC encoder an EcEncodeLayout and an EncodeRun
encode_dat_file took the Reed-Solomon shape and three block sizes as five
loose integers; they are now one Copy struct, EcEncodeLayout, which is
what Go calls ECContext. The per-row and per-batch helpers took the same
six sinks and the offsets; they become methods on EncodeRun, which owns
the borrows for one run, so each call names only the offset and block
size that vary. The byte-level work is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: describe a .dat rebuild with DatRebuild instead of nine arguments
write_dat_file_from_shards, its _with_dirs twin and the private
write_dat_file were three layers over one nine-argument signature. One
public function now takes a DatRebuild, whose shard_dirs is None when
every shard sits beside the .dat and Some(dirs) for the cross-disk
reconciled layout. The field docs carry what the function doc used to
say about the encode-time size and the block layout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: split copy_file_from_source's fifteen arguments into two structs
CopyFileSpec is the per-file request (what to ask the source for, where
it lands, whether its bytes count as progress); CopyProgress is the
sender, throttler and report state that all three files of one
VolumeCopy share, held by &mut across the calls. The three production
call sites now read as the .dat/.idx/.vif literals they are, instead of
positional trues and falses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: create volumes from a VolumeSpec
Volume::new, DiskLocation::create_volume and Store::add_volume each
took the same five-value tail of Go's NewVolume argument list:
collection, replica placement, TTL, preallocation and needle version.
That tail is now VolumeSpec, a Copy struct whose Default is what almost
every test wanted anyway (empty collection, no replication, no TTL, no
preallocation, current version), so most of the 104 call sites shrink
to `&VolumeSpec::default()` or name the one field they set. The id,
directories, index kind and disk type stay positional because they
differ at every site.
Two imports that only test modules use moved into those modules, and
DiskLocation no longer imports ReplicaPlacement.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* pb: add multipart concurrency fields to RemoteConf and tier move requests
RemoteConf gains upload_concurrency/download_concurrency (0 = client
default); VolumeTierMoveDatToRemote/FromRemote requests gain a
concurrency field (0 = backend default).
* remote storage: honor RemoteConf upload/download concurrency in s3 and azure clients
s3 client: ReadFile passes conf download_concurrency to the downloader,
WriteFile uses upload_concurrency for the uploader; previously
hard-coded 1 upload / 5 download parts. 0 keeps defaults. Same for
azure client.
* storage: plumb concurrency through backend interface and tier upload/download
BackendStorage.CopyFile/DownloadFile take a concurrency hint (<=0 =
backend configured default); s3 backend reads
upload_concurrency/download_concurrency from scaffold config with
parseConcurrency fallback, rclone updated to the new signature. Tier
move gRPC handlers forward the request concurrency to the backend.
* shell: -upload_concurrency/-download_concurrency for remote.configure, -concurrent for volume.tier
remote.configure exposes upload/download concurrency persisted into
RemoteConf; volume.tier move/evict commands forward -concurrent to the
tier move requests. Documented in master-cloud.toml scaffold.
* test: cover concurrency propagation in remote tier integration test
* remote.configure: merge existing config on partial update
Load the stored RemoteConf before saving so a partial update (e.g. only
-upload_concurrency) preserves credentials, endpoints, and type instead
of replacing them with new-config defaults. Only treat a confirmed
ErrNotFound as a new configuration; propagate all other load errors so a
transient filer failure does not overwrite stored settings.
On a type transition, reset backend-specific fields to the destination
type's new-config defaults rather than inheriting the old backend's
empty values. Bound configured concurrency to a sane maximum.
* remote storage: honor configured download concurrency in S3 and Azure
ReadFileWithConcurrency now resolves a zero request override against the
client's configured download_concurrency (new downloadConcurrency()
helpers), so the remote-mount/cache read path honors
RemoteConf.DownloadConcurrency instead of the hard-coded default.
Azure also clamps the resolved value to math.MaxUint16 regardless of
whether the fallback was used, preventing uint16 wraparound when a
configured value exceeds 65535.
* shell: rename -concurrent to -concurrency and validate tier transfer bounds
Rename the -concurrent flag to -concurrency across volume.tier.upload,
volume.tier.download, and volume.tier.compact to match the proto field and
RemoteConf field names. Add validateTierConcurrency to reject values that
would wrap int32 or exceed a 1024 cap before constructing the request.
* server: clamp tier move concurrency in gRPC handlers
Add clampTierConcurrency to both VolumeTierMoveDatToRemote and
VolumeTierMoveDatFromRemote handlers so a direct gRPC caller cannot spawn
an unbounded number of network workers.
* trim verbose comments added with concurrency feature
Remove redundant doc comments on the backend interface, rclone backend,
s3_backend parseConcurrency, and test helpers that restated the obvious.
* remote.configure: apply type defaults before re-parse so explicit flags win
applyTypeDefaults ran after the second flag parse, overwriting explicit
destination flags (e.g. -s3.region=eu-west-1) with new-config defaults.
Move the type-transition default reset before the re-parse so user-supplied
flags override the destination defaults.
* remote.configure: only treat explicit -type as a type transition
The first parse defaults -type to s3, so a concurrency-only update on an
existing non-S3 config captured requestedType=s3 and wrongly triggered a
type transition, resetting the stored backend to S3. Use fs.Visit to
detect whether -type was explicitly supplied; an omitted -type keeps the
stored backend.
---------
Co-authored-by: Jack Meredith <9480542+jackusm@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: preserve versions for POST policy uploads
Route POST policy uploads through the existing version-aware write helpers
and validate promoted Object Lock headers before writing.
Return the generated version ID when versioning is enabled, return
x-amz-version-id: null when versioning is suspended, and omit the header
when versioning has never been enabled.
* s3: reuse versioning helpers in POST policy handler
Route the POST policy handler through the existing getVersioningState
and isObjectLockEnabled helpers instead of open-coding the object-lock
forces-versioning-enabled rule, matching the PUT path.
Drop the x-amz-version-id: null response header for suspended
versioning; the PUT handler omits it and the S3 PutObject sample
response for suspended buckets does not include it. Trim the moved
fileSize comment.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* build(deps): bump github.com/redis/go-redis/v9 from 9.21.0 to 9.22.0
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.21.0 to 9.22.0.
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.21.0...v9.22.0)
---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
dependency-version: 9.22.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* test(redis_conf): track go-redis 9.22.0 default read timeout of 5s
go-redis 9.22.0 raised the default ReadTimeout from 3s to 5s (part of the
cross-SDK configuration alignment). Update TestUnsetKeepsGoRedisDefaults to
expect the new default so the bump in #11306 stops failing CI.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: grant List to the generated read-only S3 identity
The chart's anvReadOnly identity only carried the Read action, so its
credentials could GetObject and HeadObject but every ListObjects request
was denied: List is a separate action and the identity check is an exact
match. Add List so the read-only credentials can list buckets and objects.
Writes stay denied. Update the README example to match.
Bump the chart to 4.47.1. The chart label is part of the s3 and
all-in-one pod templates, so the upgrade rolls the gateways and they
reload the identity config, which is only read at startup.
Fixes#11317
* helm: roll standalone S3 and all-in-one on s3 config changes
Mirror the filer checksum/s3config pod annotation in the standalone S3
and all-in-one deployments so a changed generated S3 secret triggers a
rollout during a normal helm upgrade without relying on a chart version
bump.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: populate Initiated timestamp in ListMultipartUploads
ListMultipartUploads returned each upload with only Key and UploadId,
omitting the Initiated timestamp. Clients such as GeeseFS rely on this
field to expire stale uploads and crash on its absence. Set Initiated
from the upload directory entry creation time so repeated listings
preserve the original initiation time.
* test/s3: verify Initiated timestamp in ListMultipartUploads
Add an integration test that initiates a multipart upload, lists it,
and asserts the Initiated field is populated and preserved across
repeated listings rather than reflecting the listing time.
* admin: extract isFlagExplicitlySet helper from applyViperFallback
No behavior change; the inline flag-visit check becomes a reusable
helper so the upcoming bind-address default can share it.
* admin: default to 0.0.0.0 for authenticated HTTP, keep worker gRPC on loopback
PR #11185 made the admin HTTP server default to 127.0.0.1 to stop
exposing the unauthenticated admin API on the network by accident.
That also locked out operators who already authenticated with
-adminPassword: their UI became unreachable from the network after
upgrade unless they added -ip=0.0.0.0 (see #11303).
An authenticated deployment is safe to expose, so auto-upgrade the -ip
default to 0.0.0.0 when -adminPassword or [https.admin] mTLS is
configured. The loopback default stays for the unauthenticated case, so
the unauthenticated API is never exposed on the network. An explicit
-ip is always honored.
The worker gRPC control plane has no password auth (only mTLS), so it
must not follow the HTTP upgrade. Give it a separate bind address that
stays on loopback unless -ip is explicit, so adminPassword no longer
re-exposes the unauthenticated worker stream.
* admin: hint loopback-only bind in startup banner
When the admin server binds to loopback (the default for the
unauthenticated case), print a one-line hint that it is not reachable
from other hosts and how to expose it. This helps operators who, after
the #11185 loopback default, can no longer reach the UI from another
machine quickly see the cause and the fix without reading the docs.
* admin: keep worker gRPC on loopback, decouple from https.admin mTLS
The worker gRPC auto-upgrade to 0.0.0.0 was gated on hasMTLS, which
reads the https.admin (HTTP) mTLS config. The worker gRPC mTLS comes
from grpc.admin + grpc.ca, a separate config, so:
- https.admin mTLS without grpc.admin mTLS widened the worker gRPC to
0.0.0.0 unauthenticated (re-exposing the control plane), and
- grpc.admin mTLS without https.admin mTLS left the worker gRPC on
loopback, blocking authenticated remote workers.
Drop the worker gRPC auto-upgrade entirely. The worker gRPC keeps the
raw -ip value (loopback by default), matching the pre-existing
behavior; an operator who wants remote workers sets -ip explicitly.
Only the HTTP admin listener auto-upgrades to 0.0.0.0 when
authenticated.
Addresses review feedback on #11314 from Devin and Greptile.
* s3api: add TrustedProxies allowlist helper for aws:SourceIp extraction
Introduces a policy_engine.TrustedProxies type that parses a
comma-separated list of bare IPs and CIDRs (mirroring Guard.UpdateWhiteList)
and extracts the client IP for aws:SourceIp condition evaluation.
When the direct TCP peer is in the allowlist, X-Forwarded-For is walked
right-to-left skipping trusted hops (then X-Real-Ip); otherwise the direct
peer address is returned. This is the building block for restoring
configurable forwarded-header trust removed in b88156f (#11231), as
proposed in #11302.
* s3api: honor trusted-proxy allowlist in bucket/IAM policy engine
Make ExtractConditionValuesFromRequest a method on *PolicyEngine so it
can use the engine TrustedProxies when resolving aws:SourceIp. With no
allowlist configured the behavior is unchanged from b88156f: the direct
TCP peer is used and forwarded headers are ignored. When an allowlist is
configured via SetTrustedProxies, requests from a trusted peer honor
X-Forwarded-For (right-to-left) then X-Real-Ip.
Update the two call sites (auth_credentials.go, s3api_bucket_policy_engine.go)
and the engine tests to the method form, and add a regression test for the
trusted-proxy path.
* s3api: honor trusted-proxy allowlist in IAM role/session policies
Make extractRequestContext and extractSourceIP methods on
*S3IAMIntegration so they can use the integration TrustedProxies when
resolving aws:SourceIp. With no allowlist configured the behavior is
unchanged from b88156f: the direct TCP peer is used and forwarded
headers are ignored. When an allowlist is configured via
SetTrustedProxies, requests from a trusted peer honor X-Forwarded-For
(right-to-left) then X-Real-Ip.
Update the call site in isActionExplicitlyDeniedByIAM to type-assert
the integration and use the method, and add a regression test for the
trusted-proxy path.
* s3api: load [s3.trusted_proxies] from security.toml and wire to engines
Read s3.trusted_proxies.white_list (comma-separated IPs/CIDRs) from
security.toml and propagate the allowlist to the bucket policy engine,
the IAM policy engine (persisted across rebuilds via
IdentityAccessManagement.SetTrustedProxies), and the IAM integration.
Reloaded on SIGHUP alongside the JWT signing keys. Document the new
section in the scaffold security.toml.
Closes#11302.
* s3api: harden TrustedProxies parsing and X-Forwarded-For traversal
Canonicalize bare IP entries (via net.ParseIP + String) so non-canonical
IPv6 allowlist entries such as 2001:0db8::1 match peers rendered as
2001:db8::1, and log+skip unparseable bare entries instead of storing
them inertly.
When walking X-Forwarded-For right-to-left, stop at the first malformed
(non-empty, unparseable) entry instead of skipping it, and only fall
back to the leftmost valid IP when the chain was well-formed. This
prevents a malformed hop from masking a forged IP to its left.
Addresses review feedback on #11315.
* s3api: make TrustedProxies reload race-free via atomic.Pointer
Store the trusted-proxy allowlist behind sync/atomic.Pointer in
PolicyEngine and S3IAMIntegration so SIGHUP reloads (which swap the
allowlist) cannot race with concurrent request handlers reading it.
This mirrors the existing Guard guardState pattern. The
IdentityAccessManagement copy is already protected by iam.m.
Addresses review feedback on #11315.
* shell: warn when fs.mergeVolumes source holds only orphan needles
fs.mergeVolumes traverses filer entries, so a source volume whose
needles are all orphans — filer entries lost to a crashed write or a
wiped filer store — produces only the plan header and exits 0: no move,
no skip, no error. Operators read that as a successful merge while the
real cleanup (volume.fsck) never runs, and dat>idx volumes keep coming
back read-only after restarts.
Count the source-volume needles seen during traversal and, when a plan
source was never seen but its index still reports needles, print a
warning pointing at volume.fsck. Dry-run warns too.
* shell: make needle counting concurrency-safe and count manifest sub-chunks
TraverseBfs runs its callbacks from five workers, so the plain
needlesSeen map raced between source-heavy merges (fatal concurrent
map writes). All increments now funnel through a mutex-guarded
recordSeen closure.
Manifest sub-chunks that live on planned source volumes are now
recorded too — rewriteManifestChunk visits them (including dry-run
and capacity-skipped ones) but previously never marked their source,
which produced false 'orphan needles' warnings for sources whose
chunks were all reached through manifests.
* shell: extract sourceNeedleCounter so the concurrency test covers the production path
The orphan-warning recording was a closure local to Do, so
TestWarnUnreferencedSources_ConcurrentRecording could only exercise a
test-local copy of it — a regression in the production mutex would pass
the test. Lift the map and mutex into a sourceNeedleCounter type with
record/count methods and use it from Do and the test, so the -race test
now drives the actual recording path. Trim the verbose comments added
with the warning while here.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust: apply clippy --fix to both crates
The mechanical part of a clippy sweep: `cargo clippy --all-targets --fix`
on seaweed-volume and the seaweed-worker workspace, hand-reviewed. Both
manifests declare their MSRV (1.91.1 and 1.94.1), so every suggestion
clippy applied is within it: the collapsible_if sites become let chains
(1.88, edition 2024), `% n == 0` becomes is_multiple_of (1.87),
chunks_exact with a constant becomes as_chunks (1.88), repeat().take()
becomes repeat_n (1.82), and io::Error::new(Other, ..) becomes
io::Error::other (1.74). The rest is redundant clones, borrows, casts,
closures and field names.
Nothing here changes behaviour. The three let_and_return sites in
needle_map.rs and store_ec.rs deserve a note: the `let result = ..;
result` shape was a deliberate edition-2021 workaround to drop a redb
guard before the table it borrows. Edition 2024 drops tail-expression
temporaries before locals, which is why clippy now flags it, and the
two comments that described the workaround say so instead.
Manual edits on top of the tool output: the blocks clippy rewrote are
re-indented the way rustfmt lays them out (only those blocks — the
crate is not rustfmt-clean and a whole-crate fmt would bury this diff),
the blank lines let_and_return left behind are removed, and the CRC
legacy_value test compares against a literal worked out from the
original shift formula rather than restating rotate_right.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust: clear the clippy warnings --fix cannot apply, and say why the rest stay
Hand fixes for the lints clippy only reports. Behaviour is unchanged
throughout; each rewrite is the one clippy names.
- needless_range_loop (7): index loops over shard vectors become
iterator loops. Where the old code indexed `v[..n]` the new loop
iterates `v[..n]` so an undersized vector still panics the same way.
- field_reassign_with_default (6): struct literals with `..Default`.
- redundant_pattern_matching (3): `if let Err(_) = guard.check()` becomes
`.is_err()`, which also releases the read guard at the end of the
condition instead of at the end of the block.
- manual_strip (2), manual_checked_ops, format_in_format_args,
redundant_locals, wrong_self_convention (to_vif takes self by value,
so it is into_vif; CompactEntry is Copy, so to_needle_value takes self).
- type_complexity (2): `OrphanShardLoad` and `RawNeedleEntry` name two
tuples that were spelled out inline.
- new_without_default: CompactNeedleMap gets a Default that calls new().
- suspicious_open_options: a test helper spells out `.truncate(false)`,
which is what `.create(true).write(true)` already did.
What stays, and the attribute that says so:
- too_many_arguments (10): `#[expect]` on each function. Folding 8–15
parameters into a struct is a design change, not a lint fix.
- await_holding_lock / readonly_write_lock: one test holds the store
write guard across a sleep on purpose, as a barrier that parks the
copy task at the mount block. `#[expect(.., reason = ..)]` records it.
- module_inception: needle/needle.rs mirrors the Go package layout.
Two lints become crate-wide policy in `[lints.clippy]`, with the reason
next to each: result_large_err, because every RPC path returns
tonic::Status (176 bytes) and boxing it would change every handler
signature; and needless_update, because `..Default::default()` on a
protobuf message literal is what lets a proto gain a field without
touching every constructor (all 11 sites are pb messages). The worker
workspace gets the same table and its members opt in with
`lints.workspace = true`; its generated plugin.rs also allows
large_enum_variant on prost's oneof enums.
Both crates are now clean under `cargo clippy --all-targets -- -D warnings`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust volume: use the std APIs the 1.91 MSRV already pays for
The crate declares rust-version 1.91.1, so a few things the code still
worked around are plain std now. All of them come from the 1.85–1.91
release notes; nothing here needs a newer toolchain than the manifest
already requires.
- std::sync::LazyLock (1.80) replaces the lazy_static! block in
metrics.rs, and the lazy_static dependency goes. Every use site reads
the same through Deref, so no caller changes.
- Duration::from_mins / from_hours (1.91) replace `from_secs(v * 60)`
and `from_secs(v * 3600)` in the option parser and the shard-location
refresh TTLs. One difference for the parser: an absurd count that
overflows u64 seconds now panics in release builds too, where the
multiplication used to wrap.
- Result::flatten (1.89) replaces `.and_then(|r| r)` on the replication
join handle.
- OsStr::display (1.87) replaces `to_string_lossy()` where the name was
only being formatted; the output is byte-identical.
- `#[allow]` becomes `#[expect]` (1.81) on the suppressions that are
meant to be permanent, so a suppression that stops being needed
becomes a warning rather than lingering. Doing that found four that
already had: dead_code on ChunkManifest, base_name and last_io_error,
and too_many_arguments on read_from_data_shards, which is down to
seven parameters. Those attributes are deleted. The three allows that
depend on cfg (a unix-only mutation, a linux-only field set, a
profiling-only parameter) stay as allow, because expect would be
unfulfilled on the other platforms.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* ci: add a commented-out clippy step to both Rust workflows
Both crates are warning-free under `cargo clippy --all-targets
-D warnings` now. Whether that becomes a gate is a policy call, so the
step is present but commented out; uncommenting it is the whole change.
The comment points at the `[lints.clippy]` table where crate-wide
exceptions are recorded, so the gate does not become a reason to
sprinkle allows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust volume: guard parse_duration against overflow panics
Duration::from_mins/from_hours panic when the count overflows u64
seconds. Use checked_mul so an oversized CLI value falls back to the
parser default instead of crashing volume startup.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* admin: bind worker gRPC listener to -ip instead of wildcard
The worker/plugin gRPC control plane called net.Listen("tcp", ":port")
directly, so it wildcard-bound every interface and ignored the -ip setting.
A cluster bound to loopback still exposed the unauthenticated
WorkerService/PluginControlService streams on 0.0.0.0. Bind through
util.JoinHostPort(bindIp, port) so the listener honors -ip like the
master, filer, and volume gRPC listeners.
* admin: warn when worker gRPC is exposed off loopback without mTLS
The worker gRPC stream has no password auth, so grpc.admin mTLS is the
only effective control once the listener leaves loopback. An operator who
sets -adminPassword and binds -ip=0.0.0.0 authenticates the HTTP API but
still exposes the unauthenticated worker control plane. Log a startup
warning naming the port and the mTLS knobs so the exposure is not silent.
* admin: address review on worker gRPC bind fix
- mini: reserve the admin gRPC port with util.JoinHostPort so an IPv6
bindIp (e.g. ::1) does not form an invalid unbracketed address and
lose the reservation.
- worker gRPC: track whether grpc.admin mTLS credentials actually loaded
rather than only whether they were configured, and gate the
non-loopback exposure warning on that. A cert/key that fails to load
now still warns instead of silently suppressing.
seedAndSpread() calls `volume.grow -dataNode X` without pinning the rack.
The master's grow picks the rack by weighted-random when -rack is unset,
and only one of the three racks holds the requested data node, so an
unpinned grow lands on the wrong rack two times out of three. The
VolumeGrow RPC swallows the "No matching data node" failure for
non-cache collections, so those grows count as success without creating
a volume. The per-server cap (maxGrowsPerServer=4) is then exhausted by
silent no-ops before the volumes ever spread, and seedAndSpread times
out with "volumes never spread across >=2 disks on all 3 nodes".
Pin -dataCenter dc1 and -rack rack{i} alongside -dataNode so every grow
reaches the target node. This removes the timing-sensitive assumption
that made TestECVacuumDuplicateShardClaimAcrossDisks flaky.
* rust-volume: fix Windows build of find_needle_from_ecx
The .ecx binary-search fallback path used on non-Unix targets
(Seek + Read, both &mut self receivers) requires the ecx_file
binding to be mutable. On Unix the read_exact_at path takes &self,
so the mut would be unused there — gate that warning with
#[cfg_attr(unix, allow(unused_mut))].
Without this the build-rust-volume-windows CI job fails with
E0596 at ec_volume.rs:1033, breaking the weed-volume_windows_amd64
release asset.
* rust-volume: use positional seek_read for .ecx lookups on Windows
The previous fix (making ecx_file mut) compiled but left the Windows
fallback using Seek + Read on the shared .ecx file cursor. Concurrent
find_needle_from_ecx calls could interleave seek/read and read the
wrong index entry, corrupting the binary search (raised by Devin and
Greptile review on the PR).
Switch the Windows path to std::os::windows::fs::FileExt::seek_read,
which is positional (offset passed via OVERLAPPED, cursor untouched)
and takes &self — so the binding no longer needs mut, and concurrent
callers on the cached handle can't interfere. Mirrors the existing
read_exact_at helper in storage::volume.
Add a compile_error fallback for non-unix/non-windows targets to
match the convention in storage::volume.
filer: clean up manifest resolve error propagation and add webdav test (#78)
Drop GitHub issue references from comments and trim verbose comments.
Replace the viewFromChunksOrErr helper with the existing
NonOverlappingVisibleIntervals + ViewFromVisibleIntervals at the stream
call sites, and add a WebDavFile.Read regression test for the manifest
resolution failure path.
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* filer: add guarded remote-storage client builder hook for lazy fetch
The lazy-remote fetch path (maybeLazyFetchFromRemote) resolved its
remote-storage client through the unguarded shared cache, bypassing the
SSRF chokepoint (BuildGuardedRemoteStorageClient) that the CVE-2026-73080
remediation wired into the volume, filer stream and s3 stream dial paths.
Add a RemoteStorageClientBuilder hook on Filer plus conf-only lookups on
FilerRemoteStorage, and route the lazy fetch through the builder when set
(endpoint deny-list + DNS-rebinding-safe dialer), falling back to the
shared cache otherwise. The filer server wires the builder in a follow-up.
* filer: route lazy directory listing through the guarded remote client
maybeLazyListFromRemote shared the unguarded client resolution of the
fetch path, so a caller-supplied remote endpoint was dialed without the
SSRF deny-list or rebinding-safe dialer. Resolve the conf and build the
client through buildRemoteStorageClient so the same guard covers listing.
* filer: route lazy remote delete through the guarded remote client
maybeDeleteFromRemote issued outbound DELETE/RemoveDirectory requests
through the unguarded client, giving a write-side SSRF to a caller-chosen
endpoint. Resolve the conf and build the client through
buildRemoteStorageClient so the endpoint deny-list and rebinding-safe
dialer apply to the delete path as well.
* filer server: wire the guarded remote client builder into the filer
Set Filer.BuildGuardedRemoteClient to BuildGuardedRemoteStorageClient and
forward AllowUntrustedRemoteEndpoints so the lazy-remote fetch, list and
delete paths apply the same SSRF endpoint checks as the volume and
streaming read paths.
* filer: test lazy fetch honors the guarded remote client builder
Add a regression test that sets BuildGuardedRemoteClient to a rejecting
builder and asserts maybeLazyFetchFromRemote returns no entry without
reaching the remote, covering the SSRF guard wired in the prior commits.
* filer: skip remote client for local-only lazy deletes
maybeDeleteFromRemote resolved and validated the mount's remote client
before checking entry.Remote, so a local-only file (no Remote entry) under
a mount whose endpoint the guard rejects failed to delete: the guard
error aborted the metadata deletion, leaving a file that needs no remote
operation undeletable. Move the local-only check ahead of client
construction so only remote-backed files and directories pay the guard.
* filer: build the guarded remote client inside the lazy singleflight
The lazy fetch and list paths built the guarded client before their
singleflight blocks, so concurrent requests for the same key each
allocated a fresh SDK client and HTTP transport even though only one
remote operation ran. Move client construction inside the singleflight
so the deduplicated operation builds it once, matching the per-request
guard semantics of the sibling streaming paths without the duplicate
transport churn.
* filer: test guarded rejection for the lazy list and delete paths
Add regression tests that set BuildGuardedRemoteClient to a rejecting
builder and assert the lazy list does not reach the remote, a
remote-backed file delete is blocked, and a local-only file under a
rejected mount still deletes (covering the local-only fix).
* filer: decouple lazy guarded-client build from the first caller's context
Building the guarded client inside the singleflight made concurrent
fetches share the first caller's context. If that caller canceled while
endpoint DNS validation was running, the builder returned an error and
published a not-found result to other callers whose contexts were still
valid. Build with context.WithoutCancel so the guard's DNS validation
is not tied to any single caller's cancellation, matching the list
path's existing decoupling for the remote operation itself.
* filer: reject remote-storage confs that dial blocked endpoints at load
The filer's lazy-fetch / lazy-list / remote-delete paths resolve remote
storage clients by name from FilerRemoteStorage.storageNameToConf and
dial them via remote_storage.GetRemoteStorage, which bypasses the SSRF
deny-list the volume server (BuildGuardedRemoteStorageClient) and the
filer's own direct-read path apply. A RemoteConf planted under
/etc/remote with a loopback / private / IMDS S3 endpoint is reloaded into
storageNameToConf on the next metadata-change event and then dialed on
the next cache miss — server-side request forgery from the filer.
Apply the volume server's SSRF deny-list at conf load time, the single
chokepoint that populates storageNameToConf:
- Add RemoteStorageConfValidator, injected into FilerRemoteStorage by
the filer server (the filer package cannot import the server package).
A conf that fails validation is dropped from storageNameToConf, so the
name-based client resolution on the lazy paths returns "not found"
instead of dialing the blocked endpoint.
- Add ValidateRemoteConfForLoad in weed_server, which mirrors
BuildGuardedRemoteStorageClient's gcs credential + endpoint checks
(validateRemoteEndpoint via guardedRemoteClient) without building a
client. allowUntrusted skips the check, mirroring the volume server
opt-out (-filer.allowUntrustedRemoteEndpoints).
- The filer server injects the validator at construction.
A conf whose type dials a fixed provider host (no caller-supplied
endpoint) passes; only caller-influenced endpoints are denied.
* filer: skip DNS resolution in the load-time SSRF validator
ValidateRemoteConfForLoad resolved hostnames during /etc/remote reload,
so a transient DNS failure (2s timeout) dropped the conf from the fresh
map that replaces the live map, disabling a working mount until the next
metadata event. The build-time guard (BuildGuardedRemoteStorageClient)
already re-resolves and re-validates the endpoint at dial time with the
rebinding-safe dialer, so DNS at load is redundant for security.
Split the static checks (scheme, IMDS hostnames, IP-literal blocked
addresses, gcs credentials) into validateRemoteEndpointForLoad, which
does no DNS. Hostname endpoints pass at load and are caught at dial if
they resolve to a blocked address. This preserves fail-fast for
statically-blocked confs (loopback IPs, IMDS hostnames) without letting
transient DNS failures disable mounts.
* filer: accept empty S3 endpoints in the guarded remote client builder
guardedRemoteClient returned ok=true with an empty endpoint for a
standard AWS S3 config (no custom S3Endpoint), so
BuildGuardedRemoteStorageClient and ValidateRemoteConfForLoad rejected
it with "remote endpoint is empty" — breaking standard AWS S3 mounts on
the lazy paths and the sibling streaming read paths that already use the
guarded builder.
An empty endpoint is not caller-supplied: the AWS SDK derives the
regional endpoint from the region, so there is nothing for the SSRF
guard to validate. Return ok=false for empty S3-compatible endpoints so
the builder falls through to the shared unguarded cache, matching the
historical behavior for standard AWS S3.
* filer: accept gcs credentials file paths in the guarded remote client builder
checkGcsCredentials rejected all filesystem paths, so a gcs mount
configured with remote.configure -gcs.appCredentialsFile (which stores
a path in GcsGoogleApplicationCredentials) was rejected by
BuildGuardedRemoteStorageClient with "gcs credentials must be inline
JSON". This broke existing gcs mounts on the volume, filer, and s3
remote-mount read paths that use the guarded builder.
Read and validate the file content instead of rejecting the path,
mirroring what the gcs client itself does in MakeWithHTTPClient. A path
that does not exist or does not contain valid gcs credentials is still
rejected before any client is built. guardedRemoteClient now reads the
file to extract the token exchange URL for the SSRF deny-list, so the
rebinding-safe dialer still guards the token endpoint.
* filer: resolve gcs credential paths and avoid leaking file existence
loadGcsCredentialsContent passed the raw credentials string to os.ReadFile,
so a documented ~/path (as written by remote.configure
-gcs.appCredentialsFile=~/...) was rejected because os.ReadFile does not
expand ~. It also wrapped the os.ReadFile error, which includes the
file path, exposing file existence to a caller who planted a conf with
an arbitrary path.
Resolve the path with util.ResolvePath, matching the gcs client's own
behavior in MakeWithHTTPClient. Return a generic sentinel error on read
failure so the path is not reflected in the error message. The credential
type validation still runs on the file content, so a path that does not
contain valid gcs credentials is rejected before any client is built.
* test/ec: fix EC interruption matrix slot exhaustion
The EC integration test cluster (test/erasure_coding/chaos_lifecycle_test.go)
configured each disk with -max 4 and the seedAndSpread spread loop fired
volume.grow -count 4 every 2 s with no per-server cap. Because the master
topology lags the volume.grow writes, the loop re-fired before the prior
grow was visible, over-filling disks to capacity. A full disk leaves zero
free EC shard slots (failing the cluster-wide capacity check with "no
free ec shard slots") and drops the source disk below the encode's
FreeVolumeCount >= 2 health check (failing with "no healthy replicas"),
which aborted ec.encode before any phase marker printed and made every
encode scenario in TestECInterruptionMatrix fail.
Three changes to the test cluster:
1. Raise -max from 4 to 8 per disk so the source disk always retains
FreeVolumeCount >= 2 for ec.encode's 14-shard generation (2 volume-slot
equivalents) even after the spread loop and multiple encodes.
2. Switch the spread loop from -count 4 to -count 1 so each grow lands
exactly one volume on the volume server's least-loaded disk, giving
deterministic cross-disk spreading instead of relying on a single
multi-volume grow to fan out.
3. Cap grows per server at 4 so heartbeat lag cannot run away and
over-fill disks before the master registers the prior grow.
4. Pass -minFreeSpace 0 so the test is not falsely gated by the physical
disk's free-space percentage on the host running CI (the EC shard slot
calculation separately enforces a 90 % disk-usage cap via
balancer.DiskTooFullAfter, which already guards against an over-set
maxVolumeCount on a physically full disk).
Verified locally by running TestECInterruptionMatrix twice (all encode,
decode, and balance scenarios pass, including the previously failing
encode@Deletingoriginalvolumes).
* test/ec: only count successful grows toward the spread cap
A failed volume.grow (e.g. a transient collectTopologyInfo or VolumeGrow
RPC error) would otherwise consume one of the four permitted attempts
without creating any volume, exhausting the retry budget and leaving the
loop to only poll until the Eventually timeout. Increment the per-server
counter only when commandGrow.Do returns nil.
* iceberg: confine commit/transaction/view-update write paths to authorized bucket
The create, register, and createView handlers already confine the client-
supplied metadata location to the caller table bucket and reject ".."
segments. The commit, create-on-commit, transaction, and view-update paths
read the stored metadataLocation back from the catalog and skipped the same
guard, so a location poisoned via the raw S3Tables UpdateTable API (which
persists metadataLocation verbatim) could escape the caller bucket through
a ".." segment that path.Join collapses in saveMetadataBlob.
Add confineMetadataLocation and apply it after parseS3Location on every
commit/update/transaction/view write path, mirroring the create/register/
createView check. Reject with 400 so a poisoned stored location fails the
commit instead of writing into another tenant bucket tree.
* s3tables: validate metadataLocation at the store layer
The raw S3Tables API (CreateTable, RegisterTable, UpdateTable, CreateView,
UpdateView) persisted the client-supplied metadataLocation verbatim with no
bucket-confinement or traversal check, so a caller could store a location
pointing outside its own bucket. The Iceberg REST gateway commit paths then
read that stored value back and wrote through it.
Add ValidateMetadataLocation and call it in every s3tables store handler
that accepts a metadataLocation, rejecting locations whose bucket differs
from the caller table bucket or whose path contains traversal segments. This
prevents a poisoned location from ever being persisted, complementing the
per-write-path guard added to the Iceberg commit handlers.
* iceberg/s3tables: validate location before repair and after idempotency check
Address review feedback:
- Move the commit-path confinement check ahead of repairManifests so a
poisoned stored location cannot reach manifest repair I/O before the
commit is rejected.
- Move ValidateMetadataLocation in CreateTable/CreateView to after the
existing-resource check so idempotent retries that do not consume the
requested location are not rejected for an unused bad location.
- Assert HTTP 400 in the cross-tenant reproduction tests so an unrelated
failure cannot satisfy them.
* iceberg: confine staged metadata location before load in create-on-commit
The create-on-commit path parsed the staged metadata location from the
stage-create marker and called loadMetadataFile before validating that the
staged bucket/path stay within the authorized bucket. Add the same
confineMetadataLocation guard before the read so a tampered marker cannot
direct a cross-tenant metadata read.
* iceberg/s3tables: reject bucket-only metadata locations
ValidateMetadataLocation and confineMetadataLocation accepted s3://bucket
with an empty table path. metadataDirPath then maps every such table to the
shared <TablesPath>/<bucket>/metadata directory, so tables could overwrite
or read each other's metadata files. Require a non-empty table path in both
validators; the empty-location case (where the catalog derives one) is
unaffected.
* iceberg/s3tables: reject slash-only table paths in location validation
s3://bkt/// parses to tablePath="/" which passed the empty-string check
but path.Join cleans it away, mapping to the bucket-level metadata
directory shared across tables. Update isValidTablePath to require at
least one non-empty segment and mirror the same check in
ValidateMetadataLocation, closing the gap in all callers.
* s3/lifecycle: attach admin Bearer token on internal LifecycleDelete clients
Export credential.WithS3InternalAdminAuth (renamed from withIamCacheAdminAuth)
and use it in the worker and shell lifecycle RPC adapters so lifecycle calls
carry the same admin token the IAM-cache propagation already attaches. No-op
when jwt.filer_signing.key is unset, matching the server-side checkAdminAuth.
Prepares the internal clients for the server-side auth gate that follows.
* s3/lifecycle: gate LifecycleDelete behind admin Bearer auth
Add checkAdminAuth to LifecycleDelete, matching the SeaweedS3IamCache handlers on the same internal gRPC listener (PR #11190). No-op when jwt.filer_signing.key is unset; rejects unauthenticated callers when it is. The internal worker/shell clients already attach the token in the previous commit.
* master: name the unlabeled disk layout plainly in assign errors
When no volume server serves the layout an assign targets, the error
named the empty disk type as "hdd" (HardDriveType is the empty string),
sending operators looking for servers labeled hdd when the actual
mismatch is labeled (e.g. -disk=ssd) servers versus unlabeled clients.
- describe the layout as "default (unlabeled)" when the disk type is
empty, keep %q naming for labeled types
- log the unserved-layout condition once per option instead of letting
every failing write repeat an unactionable line
Observed in production: volume servers started with -disk=ssd while CSI
mounts assign with the unlabeled layout; the per-write error stream
pointed at a nonexistent hdd fleet.
* master: bound and expire the unserved-layout warning dedupe
The dedupe map retained every distinct option key permanently. Option
keys embed request-derived fields (collection, disk type), so repeated
assignments with distinct options would grow master memory without
bound, and a retained key suppressed the warning if the same option
went unserved again after the topology recovered.
Remember last-warned timestamps instead, expiring after an hour, with a
hard cap that resets the set when a client-driven key flood fills it.
* master: silence per-retry unserved-layout log and name explicit hdd
Addresses Devin Review comments on #11290.
- The unserved-layout branch already rate-limits its warning via
assignUnservedLayoutWarning.Do, but the common epilogue still logged
lastErr at V(0) on every retry, so the flood the dedup was meant to
stop continued. Skip the epilogue log when the unserved-layout branch
owns the logging; the error is still returned to the client.
- describeDiskLayout took the canonicalized option.DiskType, but
ToDiskType folds both "" and "hdd" into HardDriveType, so an explicit
disk=hdd request was mislabeled "default (unlabeled)". Pass the
original request disk type instead: only an empty request is the
unlabeled default; an explicit hdd is named "hdd".
Adds TestAssignFailsFastNamesExplicitHdd covering the explicit-hdd
wording.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
makeupDiff called dstDatBackend.Sync() (fsync) per needle in the loop
over incrementedHasUpdatedIndexEntry. With 20000 entries in
TestLDBIndexCompaction this resulted in up to 20000 fsync calls, which
on slow CI disks exceeded the 10-minute test timeout.
Batch the sync: write all needles/tombstones first, then fsync the dat
file once in the defer alongside the existing idx fsync. The durability
guarantee is unchanged — both files are still synced before CommitCompact
writes the .cpc commit marker and swaps the files.
* cache resolved chunk manifests for Mount
* Address PR review: per-mount cache, singleflight, reuse ResolveOneChunkManifest
- Own the manifest cache per WFS mount instead of a process-global
variable, so manifests from one filer backend are never served to
another (Devin/CodeRabbit major bug).
- Coalesce concurrent cold misses via singleflight so only one fetch
runs during a cold burst (Greptile P2).
- Copy cached data after releasing the mutex so a large copy does not
block concurrent hits, inserts, and evictions (CodeRabbit nitpick).
- Reuse the existing ResolveOneChunkManifest function name instead of
introducing a new resolveOneChunkManifest wrapper.
- Validate (unmarshal) manifest bytes before caching so malformed
manifests do not poison the cache.
- Add TestChunkGroupManifestResolutionCoalescesColdMisses covering
the singleflight cold-miss path.
* Address round 2 review: coalesced-miss cancellation, test overlap
- Use singleflight.DoChan in fetchOrLoad and select on ctx.Done() so a
caller whose context is canceled while waiting for an in-flight fetch
returns ctx.Err() promptly instead of blocking for the leader's
result (Devin BUG).
- Add TestResolveOneChunkManifestCanceledWaiterReturnsDuringCoalescedMiss
covering the canceled-waiter path.
- Delay the cold-miss fixture response so the leader's fetch is still
in flight when concurrent opens join the singleflight, making the
one-fetch assertions reliable (CodeRabbit Minor).
* Address review: keep ResolveOneChunkManifest four-argument
Restore the exported ResolveOneChunkManifest to its original
four-argument signature so external callers keep compiling. Move the
cache-aware resolution into an unexported resolveOneChunkManifest
helper that accepts the per-mount ChunkManifestCache. The exported
function delegates to the helper with a nil cache, preserving the
historical uncached behavior for every non-Mount caller. The Mount
path (ChunkGroup.SetChunks) now calls the unexported helper with the
mount-owned cache. Tests and benchmarks that exercise the cache path
call the unexported helper directly.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* mount: fail reads with error when chunk manifest resolution fails
When SetChunks fails to resolve a chunk manifest (e.g. the volume is on a
remote tier with reads disabled), the sections map stays empty and
readDataAtSequential/readDataAtParallel zero-fill every missing section as
if it were a sparse hole. Reads then return all-zero data with no error,
so a plain cp of a large manifest-based file silently produces a
completely zero-filled file.
Remember the resolve error in ChunkGroup (guarded by sectionsLock) and
return it from ReadDataAt. A later successful SetChunks clears it.
Fixes the mount path of #11286.
* filer: propagate manifest resolve errors in streaming read paths
ViewFromChunks discards the chunk manifest resolve error returned by
NonOverlappingVisibleIntervals. On failure the chunk views come back
empty, and the streaming paths zero-fill the entire requested range,
serving HTTP 200 / WebDAV 200 responses whose body is all zeros.
Propagate the error in PrepareStreamContentWithThrottler,
PrepareStreamContentWithPrefetch and the WebDAV read path so these
requests fail with 500 instead.
Fixes the filer HTTP and WebDAV paths of #11286.
* mount: fail lseek with EIO when chunk manifest resolution fails
SearchChunks still consulted the stale section map after SetChunks
recorded a manifest resolution failure, so SEEK_DATA/SEEK_HOLE would
describe the unresolved regions as sparse holes or return ENXIO.
Return the recorded error from SearchChunks and map it to EIO in
Lseek.
Also add regression tests for the stream preparation error paths.
Addresses review feedback on #11287.
* storage: add Store::find_all_ec_volumes for split-disk EC lookups
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: add merge_ec_runtimes to resolve a vid's per-disk shard set
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: replace dead slots.get(14) assertion with a width-14 pin
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: build the checksum scrub plan from every per-disk runtime
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: build the local scrub plan from every per-disk runtime
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: prove the local scrub plan reaches every runtime's slots
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: make the scrub plan tests falsifiable
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: report unverifiable protection when the sidecar predates the scrubbed encode
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: commit sidecar provenance with the sidecar it describes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* volume server: scrub every disk's EC shards for CHECKSUM and LOCAL
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: run the FULL/READS parity check across split-disk shards
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* volume server: report fenced-out runtimes in FULL/READS scrubs
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: tighten verify_ec_shards ordering and missing-shard coverage
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* volume server: visit each EC volume id once in node-wide scrubs
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: cover split-disk scrub aggregation end to end
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* volume server: pin fenced-out disks and sibling-disk shards in EC scrubs
Three scrub behaviors shipped without a test at the RPC seam. Task 8
showed the seam exists, so close them here.
FULL/READS (mode 2|5) now marks a volume broken when the identity fence
excludes a runtime, where it previously reported clean. Pinned against a
control fixture whose two disks AGREE and scrub clean, so the test fails
on the clean->broken transition, not only on the message text. That needs
a structurally valid, tombstone-only .ecx (so the needle walk finds
nothing to complain about) and a seeded shard-location cache (so the
absent master does not short-circuit the scrub with an error of its own).
LOCAL (mode 3) and CHECKSUM (mode 4) now build their plans from every
per-disk runtime. Made observable by moving shard 0 -- the shard the
volume's single needle spans and the one the checksum sidecar is checked
against -- to the SIBLING disk, leaving shard 5 on the disk the singular
find_ec_volume lookup returns. Built from that disk alone, neither scrub
ever looks at shard 0.
The split-disk fixture grows a config struct rather than more positional
arguments; its defaults reproduce the existing layout byte for byte, so
the node-wide dedupe test is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: report fenced-out disks on a malformed sidecar too
`errors.extend(self.skipped)` sat below the whole status match, so only
`(Some(p), On)` ever reached it. The Invalid arm already returns a
non-empty error vector of its own, so the Go-parity contract that
silences the Off arm (`case BitrotOff: return 0, nil, nil`) does not
reach it -- appending the fence lines there costs nothing that contract
protects. A volume with BOTH a malformed sidecar and a disk the identity
fence excluded reported only the sidecar, hiding the unscanned disk
behind an unrelated integrity error.
Off stays byte-identical, and so does the `(None, On)` arm that is
documented as treating a missing payload defensively as protection off.
Off is now the ONLY status that drops the report, and the comment at the
On-path copy says so: that is the one place the parity constraint costs
us coverage.
Also corrects a false claim in the FULL/READS test's doc comment. It
said a fenced-out disk "is a disk this scrub did NOT read", which is true
only of the merge-driven parity half. The per-needle walk still resolves
`store.find_ec_volume` (store_ec.rs:281) and binds
`expected_encode_ts_ns` to that runtime (:311) -- position 0, the
EXCLUDED one on that fixture -- so `read_local_intervals`' generation
filter (:1204) makes it read the excluded disk and treat the anchor's
shards as non-local, the inverse of what `skipped` reports. The fixture's
tombstone-only .ecx walks nothing, so the test cannot tell the two apart;
the comment now says that rather than implying coverage it does not have.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: take CHECKSUM's bitrot protection from the disk that has the sidecar
`EcChecksumScrubPlan::for_volumes` read `(prot, status)` off the ANCHOR.
The anchor is the first shard-bearing runtime at the maximum `encode_ts_ns`,
chosen with no regard for which disk holds the `.ecsum`.
That sidecar is deliberately NOT mirrored across disks -- `ec_metadata_dirs()`
exists so one authoritative copy stays reachable rather than being duplicated
-- and at mount `EcVolume::new` resolves it via `load_active_bitrot_sidecar(&[])`
with no sibling directories at all; only the `VolumeEcShardsMount` RPC ever
passes `ec_metadata_dirs()`. So after EVERY volume-server restart, the
split-disk runtime that does not physically hold the sidecar mounts
`BitrotStatus::Off`. When the one copy lives on disk 1 and the anchor is disk 0,
`run()` hit `case BitrotOff` and returned `(0, [], [])`: the whole volume
scrubbed clean, silently. That is the steady state for roughly half of all
mirrored split-disk layouts, and it is the exact failure this branch exists to
remove.
Source protection from the first MERGED runtime that has any -- `On` if one
does, else `Invalid`, else the anchor's `Off`. Two facts make that safe, and
both are load-bearing:
- Every runtime that mounted `On` already passed the `geometry_matches` gate
in `load_bitrot_for_generation`, so its manifest agrees with the volume's
layout. A sidecar that contradicted it would have failed the mount.
- All merged runtimes share the same `encode_ts_ns` by construction of the
identity fence, so a sidecar from any of them describes the same encode run.
The `unverifiable_sidecar` provenance rule four lines down read
`anchor.bitrot_source_dir`; it now reads the SAME runtime `prot` came from.
Otherwise the two would describe different sidecars and the rule would vouch
for a manifest nobody is scanning against. One consequence worth naming: that
source dir is now non-empty by construction (a runtime with protection found a
file), where the anchor's was often "" and short-circuited the rule -- so on a
fenced volume whose anchor had no sidecar, an unverifiable-protection note now
surfaces where previously nothing was reported at all.
`run()` is untouched, and the `BitrotStatus::Off` arm still returns
`(0, [], [])` exactly, for Go parity with `case BitrotOff: return 0, nil, nil`.
`parity_shards` still comes from the anchor while `prot` may come from a
sibling; the geometry gate above makes them agree, and slot-width agreement is
handled separately.
The test drives mode 4 through the real RPC against a split-disk volume whose
sidecar exists only on dir1, and asserts up front that the anchor mounted `Off`
and the sibling `On` -- otherwise it would prove nothing. Reverting this commit's
one-line source change makes it report `[]` instead of `[0, 5]`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: pin the slot width, contain the shard-size fallback, and cover multi-disk FULL
Five findings from the whole-branch review, none of which changes what a
healthy volume reports.
Slot width was undefined and the two consumers disagreed (ec_volume.rs).
`merge_ec_runtimes` sizes `slots` to the WIDEST merged runtime, but the identity
fence keys on `encode_ts_ns` alone and never on geometry -- so two
same-generation runtimes whose `.vif`s disagree do merge. The mode 2|5 arm
truncates to the anchor's `data+parity` and silently drops the surplus slots,
while `EcChecksumScrubPlan::for_volumes` iterated the full width and emitted
"present but missing from sidecar manifest" for exactly those ids. Nothing in
the volume describes them -- the sidecar manifest and the Reed-Solomon matrix
are both the anchor's -- so that message was the width disagreement talking, not
a finding. The `slots` field doc now states the contract (the range is the
anchor's geometry; every consumer truncates to it) and CHECKSUM truncates.
The LOCAL `shard_size` fallback had grown a node-wide blast radius
(ec_volume.rs). `anchor.shard_file_size()` returns the anchor's FIRST held
shard, not a maximum. Before aggregation the plan read only that runtime's own
shards, so a truncated shard was contained to its disk; now that one value sizes
every merged sibling's shards, mis-offsetting `locate_data` and manufacturing
needle corruption across the node. Take the max over the merged slots, which is
how `verify_ec_shards` already answers the same question
(`if size > shard_size { shard_size = size }`). Only on the legacy
`dat_file_size == 0` path.
Multi-disk `all_local` had no end-to-end test (grpc_server.rs). The parity check
is gated on every shard being present, and the one all-local fixture keeps them
in a single directory, so every entry of `dirs` is the same string and a
permutation or off-by-one in the `slots` -> `dirs` mapping is invisible;
`test_verify_ec_shards_reads_shards_from_multiple_dirs` builds its `dirs` by
hand and never goes through `merge_ec_runtimes`. The new fixture is a real 10+4
encode split 0..=6 / 7..=13 across two store locations (the `.dat`/`.idx` stay
outside both, so `prune_incomplete_ec_with_sibling_dat` has nothing to act on),
driven through the real RPC: clean first, then a corrupted PARITY shard on the
SECOND disk -- which only the parity half can see, and only through a correct
mapping. Shifting that mapping by one, or computing `all_local` from the anchor
alone, both make it report `[]` instead of `[13]`.
Deleted `test_ec_volume_enumeration_is_deduped` (store_ec_reconcile.rs). It
built `raw` from `store.locations` and then applied its OWN inline
`filter(|v| seen.insert(*v))`, asserting on that -- a property of
`HashSet::insert`, never reaching the production dedupe. That path is covered by
`test_scrub_ec_volume_node_wide_dedupes_a_split_disk_volume`, which does fail
(2 != 1) when the dedupe is removed.
Corrected `test_verify_ec_shards_treats_a_none_dir_as_missing`'s docstring
(ec_encoder.rs). It claimed the unmounted shard "must not drag the shards that
ARE mounted down with it", but `dirs[5] = None` puts shard 5 in `broken_shards`
before the block loop, so every iteration takes the `read_failed` arm and the
parity comparison never runs: corrupting a mounted shard in that fixture changes
nothing about the result. The assertions are unchanged; the docstring now states
what they actually establish.
Also refreshed two comments that cited `shard_file_size() - 1` as the reason
`merge_ec_runtimes` prefers a shard-bearing anchor -- true before this commit,
stale after it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: correct the Fix 1 rationale and truncate the shard-size scan
The safety argument attached to `EcChecksumScrubPlan::for_volumes`'s protection
selection was false as written, and it is the argument a reviewer reads first.
`geometry_matches` compares a sidecar against the MOUNTING runtime's own
data/parity/block size, not the anchor's, and returns true vacuously when
`ec_shard_config` is `None` -- so it establishes agreement only when all merged
runtimes share one geometry, which an `encode_ts_ns`-only fence does not
guarantee and which `test_checksum_scrub_truncates_slots_to_the_anchors_geometry`
constructs a counterexample to. The second clause was weaker than stated too: a
`.ecsum` records no encode identity at all, so merged runtimes agreeing on
`encode_ts_ns` does not transfer to the sidecar.
Replace it with the property that is true, checkable from the selection itself,
and stronger for what actually matters. `anchor` is an element of `merged`, so
the `.unwrap_or(anchor)` fallback is reached only when no merged runtime is `On`
and none is `Invalid` -- in which case the anchor is necessarily `Off`. The
status can therefore only move `Off -> On`, `Off -> Invalid` or
`Invalid -> On`; never `On -> Off`, never `Invalid -> Off`. This selection
cannot stop a volume that was being scanned from being scanned, and cannot turn
a reported integrity error into silence: every change it makes is toward more
verification. The comment now also states what it does NOT establish -- geometry
agreement is not guaranteed -- and names geometry fencing as the follow-up that
would close it.
Second, `EcLocalScrubPlan::for_volumes`'s `shard_size` max scanned the FULL slot
width, violating the `slots` contract documented in the same commit that
introduced the max: the volume's shard-id range is the anchor's geometry and
every consumer must truncate to it. Pre-fix that input could not exist, because
`anchor.shard_file_size()` read only the anchor's own anchor-sized vector -- so
the max opened a new, narrow path to the same node-wide mis-sizing it exists to
close (same-generation runtimes with disagreeing `.vif`s, the wider one holding
an out-of-geometry shard larger than the in-geometry ones, `dat_file_size == 0`).
`.take(anchor.data_shards + anchor.parity_shards)` mirrors the truncation
already applied to the CHECKSUM shard scan.
The sibling `shards:` vector is left untruncated on purpose: every access in
`EcLocalScrubPlan::run` is `shards.get(sid)` with `sid < data_shards`, so the
surplus entries are inert.
No behavior change for any healthy volume, and no test added -- the suite is
unchanged at 575 passing, 0 failing, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE
* ec: aggregate split-disk runtimes in Go scrubs, mirroring Rust
Go volume scrubs previously used FindEcVolume (first runtime only), so a
volume whose EC shards are split across multiple disks was scrubbed against
just one disk's shards and the others were silently skipped. Node-wide
ScrubEcVolume also appended each disk's EcVolumeIds without deduplication,
scrubbing a split-disk volume once per disk.
Add MergedEcRuntimes/MergeEcRuntimes (Go counterpart to Rust's
merge_ec_runtimes): select the maximum EncodeTsNs as the anchor generation,
fence out runtimes whose encode generation or geometry (DataShards,
ParityShards, BlockSize) disagrees with the anchor, merge shard handles by
shard ID, and report excluded runtimes rather than dropping them. Wire it
into every scrub mode:
- INDEX: scrub the anchor's index, report skipped runtimes.
- LOCAL: aggregate local shards across all merged runtimes via a synthetic
EcVolume built from the merged shard slots.
- FULL/READS: resolve the runtime matching the anchor's encode generation
(not the first match) so the needle walk and parity phase inspect one
encode run; report skipped runtimes.
- CHECKSUM: take bitrot protection from the first merged runtime that has a
valid sidecar (On, else Invalid, else anchor's Off), preserve invalid
sidecar errors from every other merged runtime, and report skipped
runtimes.
Deduplicate EC volume IDs in node-wide ScrubEcVolume so each volume is
scrubbed exactly once.
Refactor ScrubEcVolume to share the per-needle walk via scrubEcVolumeWalk,
called by both the legacy first-runtime path and the new merged path.
Add Go regression tests covering split-disk deduplication, encode-generation
fencing, geometry fencing, sibling-disk LOCAL reach, and merge anchor
selection.
Rust: keep the previously-landed merge/fence/checksum changes intact; revert
incidental cargo-fmt drift from unrelated files so the diff stays focused.
* ec: fence merged CHECKSUM on sidecar encode generation and fix legacy shard size
Address two review findings on the Go merged-runtime scrub:
1. Sidecar provenance: a merged runtime can load a bitrot sidecar from a
sibling metadata directory (ReloadBitrotSidecar), and the merge fence may
then exclude the runtime owning that directory. Generation-0 sidecars do
not identify the encode run, so geometry validation alone cannot prove the
borrowed manifest describes the anchor shards. If the sidecar records a
non-zero EncodeTsNs that disagrees with the anchor, refuse the scan
instead of applying stale checksums to current shards and reporting false
corruption.
2. Legacy shard size: for volumes without datFileSize in .vif,
LocateEcShardNeedleInterval derives the shard size from Shards[0].ecdFileSize.
The merged shard set is compacted in shard-ID order, so a truncated
lowest-ID shard would shrink every interval and misread intact sibling
shards. Synthesize a datFileSize from the maximum mounted shard size when
the anchor lacks one, so the datFileSize>0 path uses the largest shard
size across all merged runtimes.
* ec: fix copylocks, legacy shard boundary, and encode-aware Rust lookups
Address review findings from CodeRabbit and Devin:
Go (ec_volume_merge.go):
- Remove bitrotLock copy from the synthetic EcVolume: copying a sync.RWMutex
is a go vet copylocks error. The synthetic volume uses its own zero-value
mutex; bitrot/bitrotStatus are set directly before ChecksumScrub reads them
via BitrotProtection(), so no concurrent access occurs.
- Fix legacy shard-size boundary: synthesize datFileSize from
(maxShardSize - 1) * DataShards, not maxShardSize * DataShards, to match
the legacy fallback in LocateEcShardNeedleInterval (ecdFileSize - 1). An
exact large-block boundary is ambiguous; the unadjusted size would select
an extra large row and misread intact sibling shards.
Rust (store_ec.rs):
- Add find_ec_volume_for_scrub helper that resolves by encode generation
(not first-match find_ec_volume) and use it in scrub_snapshot_under_lock,
write_back_shard_locations, and the post-refresh shard-location read.
Previously the encode-aware lookup was only used for the initial runtime
selection; the cache write-back and per-needle snapshot still used
first-match, so a split-disk volume whose first runtime was from an older
encode run would write to and read from the wrong runtime's shard-location
cache and falsely abort with 'remounted as a different encode run'.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* test(fuse_failover): dump chunk list and hex on append corruption
The failover append test (TestAppendWhileVolumeServerRestarts) failed
in CI with an 8-byte NUL region at offset 632 that appeared in both
the writer mount and the filer own view, but the failure message
only showed a quoted-string window around the divergence. That is
not enough to tell which chunk covered the zeroed bytes or which
volume server held it, so the next recurrence would be just as
unattributable.
Add a FileChunkList helper that reads the filer resolved chunk
list, and on failure dump:
- every chunk fid, offset, size, volume id, and current master
holders, flagging the chunk that covers the first divergence;
- a hex+ASCII dump of the writer mount around the divergence so
the exact zero-filled region is visible byte-for-byte.
No production code is touched; this only makes the test fail louder.
* test(fuse_failover): preserve diagnostic collection errors
Address review feedback from CodeRabbit and Greptile on PR #11285:
- FileChunkList now returns the wrapped ParseUint error when
fid.volume_id is zero and the file_id prefix is invalid, matching
FileVolumeIds instead of silently keeping vid=0 (which would
query /dir/lookup?volumeId=0 and report the wrong holders).
- dumpChunkList captures the VolumeHolders error and renders it as
'lookup failed: ...' so a failed master request is distinguishable
from a successful lookup with no holders (both previously showed
'holders=[unknown]').
- runChaosAppend captures the writer-mount read error and includes
it in the failure message so an unavailable writer view is not
mistaken for corrupted content.
MasterServer.Shutdown only stopped the Hashicorp raft implementation;
when using the default goraft backend, the raft event-loop goroutine
(leaderLoop/followerLoop) kept running after the master shut down. In
the in-process test harness this leaked goroutines across sequential
test runs, and a stale event occasionally reached a leader at term 0
and tripped the goraft "leader.elected.at.same.term" assertion,
crashing the whole test binary (CI run 34670959967, PR 11279).
Stop the goraft server in Shutdown() so its goroutines exit cleanly,
and bump seaweedfs/raft to v1.2.1 which replaces that assertion with a
graceful step-down to Follower instead of a panic.
* s3: reject virtual-host bucket retargeting via X-Forwarded-Host
SigV4 verification tries the client-supplied X-Forwarded-Host as a
signed host candidate, while routing and IAM select the bucket from
the actual Host header. A presigned URL for one virtual-host bucket
could therefore be retargeted to another bucket accessible to the same
signing identity by changing Host and adding X-Forwarded-Host.
After the signature matches a host candidate, extract the bucket that
the candidate implies (via the configured virtual-host domains) and
compare it with the bucket the router selected. Reject when they
differ, before returning success.
* test(s3api): cover virtual-host presigned URL retargeting
Add unit tests for bucketFromVirtualHost and end-to-end tests that
reproduce the X-Forwarded-Host retargeting attack for both presigned
and signed requests, plus a negative test confirming the legitimate
same-bucket case still verifies.
* s3: harden bucketFromVirtualHost for case and overlapping domains
Compare host and domain suffixes case-insensitively so a mixed-case
X-Forwarded-Host cannot bypass the consistency check. Only treat the
exact path-style domain as non-virtual-host; subdomains of a path-style
domain still match the virtual-host router pattern and must be checked.
* s3: reject list-type paired with another operation subresource
?list-type=2&ownershipControls= routes to ListObjectsV2 (the list-type
route is registered first) while the IAM action resolver resolves the
ownershipControls selector to s3:GetBucketOwnershipControls. A principal
denied s3:ListBucket but allowed s3:GetBucketOwnershipControls would
therefore list the bucket. list-type selects an operation just like the
other keys in operationSubresources, so add it there and reject the
combination before routing, matching the fix for policy&tagging (#10987).
* s3: resolve list-type to s3:ListBucket ahead of bucket subresources
The router registers the ListObjectsV2 route ahead of the bucket
subresource routes, so the action resolver should resolve list-type the
same way. Without this, a request carrying list-type and another operation
selector resolves to the subresource action (e.g. s3:GetBucketOwnershipControls)
while being served by ListObjectsV2. The ambiguity guard rejects such
combinations before routing, but resolving list-type to s3:ListBucket keeps
the resolver aligned with the router, mirroring how versions is handled.
* s3: match list-type=2 exactly in action resolver
The router selects ListObjectsV2 only for list-type=2; other values fall
through to the subresource routes. Resolve the same way so the action
matches the handler for every list-type value, not just 2.
* feat(s3api): add bucket quota S3 extension via ?seaweedfs-quota
Add a SeaweedFS-specific S3 subresource for bucket quota management:
PUT /{bucket}?seaweedfs-quota — set bucket quota (s3:PutBucketQuota)
GET /{bucket}?seaweedfs-quota — get bucket quota (s3:GetBucketQuota)
The request/response body is JSON:
{"quota_size": 100, "quota_unit": "GB", "quota_enabled": true}
Quota is stored on the bucket's filer entry (positive = enabled,
negative = disabled but retained, zero = no quota), matching the
existing admin REST API behavior. When quota is cleared, the bucket's
read-only flag is also lifted.
Authentication uses the existing S3 SigV4 flow — no new global secret
is needed. Authorization uses two new dedicated IAM permissions:
s3:PutBucketQuota
s3:GetBucketQuota
This allows integrations like Apache CloudStack to manage per-bucket
quotas through the S3 endpoint with a scoped credential, without
exposing the broad admin REST API or requiring a separate admin token.
The credential can be limited to s3:PutBucketQuota/s3:GetBucketQuota
only, preventing bucket deletion, user management, or cluster topology
changes.
The coarse-grained ACTION_PUT_BUCKET_QUOTA/ACTION_GET_BUCKET_QUOTA
constants are added to s3_constants, and the action resolver maps the
seaweedfs-quota query parameter to the fine-grained s3: actions for
policy evaluation.
* docs: update design for S3 ?seaweedfs-quota extension approach
Replace the broad admin REST API + bearer-token design with the narrow,
scoped S3 ?seaweedfs-quota extension. Update quota, usage reporting, and
SeaweedFS-side changes sections to reflect PR #11279.
* fix(s3api): address review comments on quota handler
Fix four issues identified by Devin, Greptile, and CodeRabbit reviews:
1. Integer overflow in convertQuotaToBytes: large quota_size values
(e.g. 8388608 TB) could overflow int64, wrapping to negative and
being silently treated as zero quota. Now returns an error when
size * multiplier would exceed math.MaxInt64.
2. Disabled quotas returned negative sizes in GET: the GET handler
returned entry.Quota directly, which is negative for disabled-but-
retained quotas. Now returns the absolute magnitude as quota_size
and derives quota_enabled from the sign, making the response
round-trippable.
3. Missing buckets returned 500 instead of NoSuchBucket: the PUT
handler treated all lookup failures as internal errors. Now
distinguishes filer_pb.ErrNotFound and returns ErrNoSuchBucket.
4. Trailing JSON was silently accepted: the decoder read only the
first JSON object without checking for trailing data. Now
requires EOF after the object, rejecting malformed payloads.
Also add tests for overflow detection and trailing data rejection.
* fix(s3api): cast math.MaxInt64 to int64 for 32-bit vet
On 32-bit platforms, math.MaxInt64 is an untyped int constant that
overflows int (32-bit) when used directly in fmt.Errorf with %d.
Cast to int64 explicitly to fix Go Vet 32-bit.
* docs: reconcile design doc with implementation and add AWS tools note
- Resolve open question about IAM endpoint path: driver accepts optional
iamUrl and defaults to <s3Url>/iam
- Add note explaining ?seaweedfs-quota is not callable by standard AWS tools
(aws s3api, s3cmd, rclone), and how this compares to MinIO and Ceph quota
APIs which also live outside the standard S3 API
* docs: fix IAM endpoint default — SeaweedFS IAM is at POST / on S3 endpoint
SeaweedFS registers its embedded IAM API at POST / on the same S3
endpoint (UnifiedPostHandler), not under /iam. The design doc
previously said the driver defaults iamUrl to <s3Url>/iam, which would
send IAM operations to an unregistered path. Correct the default to
s3Url.
Found by Greptile review on PR #11279.
* docs: fix credential model, signer, and GET response shape in design doc
Three issues found by CodeRabbit review on PR #11279:
1. Credential-scope contradiction: the doc claimed the service credential
is scoped to only s3:PutBucketQuota/s3:GetBucketQuota, but the
implementation uses it as the admin credential for all operations
(bucket CRUD, IAM user provisioning, quota). Document the actual
model.
2. S3Signer -> AWSS3V4Signer: the doc said 'S3Signer for SigV4 signing'
but S3Signer is legacy SigV2. Correct to AWSS3V4Signer.
3. GET response shape: the doc showed a single JSON example with 'GB'
for both PUT and GET, but GET always returns quota_unit 'B' and the
absolute byte count. Document PUT input and GET response separately.
* s3: support StringEqualsIgnoreCase and related condition operators
The S3 bucket-policy condition engine rejected StringEqualsIgnoreCase
(and StringNotEqualsIgnoreCase, StringLikeIgnoreCase,
StringNotLikeIgnoreCase), which AWS and the IAM policy engine both
accept. Add evaluators and register them in GetConditionEvaluator so
valid policies using these operators evaluate correctly instead of
being skipped.
* s3: reject bucket policies with unsupported condition operators
validateStatement did not check Condition operators, so a policy with
an unknown operator (e.g. a typo or unsupported key) was accepted at
upload time and only surfaced at evaluation, where it was silently
skipped. Reuse GetConditionEvaluator to reject unknown operators when
a policy is parsed or stored, failing closed at the entry point
instead of relying on evaluation-time handling.
* s3: fail closed on unsupported condition operators at evaluation
EvaluateConditions skipped statements whose condition operator was
unsupported, logging a warning and continuing. With no remaining
conditions to fail, the function returned true, so an Allow statement
conditioned on an unrecognized operator became unconditional and
granted access to private objects. Return false instead so an
unrecognized operator fails the condition block and the statement does
not match, matching the fail-closed behavior of the IAM policy engine.
* s3: validate condition operators at upload time only, not load time
Validating condition operators in validateStatement rejected the whole
policy document from ParsePolicy, which SetBucketPolicy uses when loading
stored bucket policies. A legacy policy saved before this change could
contain an unsupported operator, and rejecting it at load time dropped
the entire policy - including unrelated explicit Deny statements - so
the bucket lost its protections. Move the operator check into
ValidateBucketPolicy, which only the PutBucketPolicy handler and admin
UI run at upload time, so legacy policies still load and EvaluateConditions
fails the unsupported statement closed instead.
* s3: drop non-AWS StringLikeIgnoreCase and StringNotLikeIgnoreCase operators
AWS defines StringEqualsIgnoreCase and StringNotEqualsIgnoreCase but
not StringLikeIgnoreCase or StringNotLikeIgnoreCase (StringLike and
StringNotLike are case-sensitive only). Registering the wildcard
IgnoreCase variants made the engine accept operators AWS rejects. Keep
only the two AWS-defined IgnoreCase operators and add a test asserting
the wildcard IgnoreCase names are unsupported.
* s3 sink: report the source read error the SDK hides
filer.backup stops for good on an event whose chunks are gone from the
volume servers: the uploader reads the body, the read fails with the
volume's 404, and the AWS SDK returns "ContentLength=N with Body length 0"
without the cause. isIgnorable404 would skip such an event, but it never
sees the 404, so the event is retried forever and the checkpoint never
advances.
ChunkStreamReader keeps its first source failure and the s3 sink returns
it when the upload fails.
* s3 sink: trim verbose comments on source error propagation
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(vacuum): stop comparing compact size against the live needle map
CompactByIndex's post-copy integrity check compared bytes written to
the .cpd against v.nm.ContentSize()-DeletedSize(), the live map that
keeps mutating for as long as the volume stays writable during the
copy. Any write landing after the point-in-time index snapshot was
loaded made the live map's tally exceed what got copied, aborting
compaction with "unexpected new data size" — even though
CommitCompact's makeupDiff exists specifically to reconcile writes
that land mid-copy. On a busy volume this can fail every vacuum cycle.
Tally the expected live size from oldNm, the same frozen snapshot the
copy loop reads from, instead of the live map. This keeps the check's
original protection (destination smaller than what should have been
copied signals real data loss) while removing the false positive from
ordinary concurrent traffic.
* fix(vacuum): stop double-subtracting skipped bytes from the size check
Unreadable needles return before reaching the expectedLiveBytes tally,
so it already excludes them. Subtracting skippedDataBytes again on top
loosened the integrity check's margin by that same amount, letting a
.cpd short of the true expected size slip past undetected — the exact
failure mode the check exists to catch. Flagged independently by three
automated PR reviewers (Devin, Greptile, CodeRabbit).
Extract the comparison into exceedsExpectedCompactedSize and drop the
subtraction entirely; add TestExceedsExpectedCompactedSize to pin the
threshold to expectedLiveBytes alone.
* fix(vacuum): trim verbose integrity-check comment
Reduce the 8-line block comment to a concise 3-line rationale. No
behavior change.
* fix(vacuum): mirror compact integrity check in Rust volume server
Mirror the Go fix in the Rust volume server's do_compact_by_index:
tally expected_live_bytes from the frozen index snapshot (not the live
needle map) and compare the compacted .dat against it after the copy.
Unreadable needles already return before the tally, so no skipped-byte
adjustment is needed. Adds exceeds_expected_compacted_size and two
regression tests.
* fix(vacuum): exercise makeup_diff in Rust concurrent-write test
Address CodeRabbit review: write a needle after compaction (before
commit), then call commit_compact() and assert the late write survives
via makeup_diff. This actually exercises the concurrent-write path
rather than just confirming the integrity check passes.
---------
Co-authored-by: chrislusf <chris.lu@gmail.com>
* test: reproduce expired directory cache degrading lookup to N RPCs
After cacheMetaTtlSec elapses the kernel can still serve a directory
listing from its page cache, so ReadDir never runs and EnsureVisited is
not called. Metadata lookups then fall through to one LookupEntry RPC
per entry instead of rebuilding the directory cache once.
Issue #11262
* mount: add expired-directory rebuild predicate with cooldown to InodeToPath
ShouldRebuildExpiredDir distinguishes a TTL-expired cached directory from
a never-cached, invalidated, evicted, or read-through one (those clear
isChildrenCached, while a plain TTL expiry keeps it set). It also gates
retries on a cooldown since the last failed rebuild attempt, recorded by
MarkRebuildAttempt, so a transient listing failure does not trigger a
full rebuild on every later lookup.
Issue #11262
* mount: rebuild expired directory cache on entry lookup
When the kernel still serves a directory listing from its page cache past
cacheMetaTtlSec, ReadDir never runs and EnsureVisited is not called, so
lookupEntry issues one LookupEntry RPC per entry. Rebuild the expired
directory once via ensureDirectoryVisited before the cache-hit check so
later lookups are served locally. The EnsureVisited singleflight
deduplicates concurrent rebuilds.
On a non-oversized rebuild failure, record the attempt so the cooldown
suppresses repeated rebuilds while the listing keeps failing; once it
elapses a later lookup retries, recovering without waiting for ReadDir.
Oversized dirs are already marked read-through by ensureDirectoryVisited.
Issue #11262
* test: cover concurrent rebuild dedup and rebuild-cooldown fallback
Add a test that runs concurrent lookups into the same expired directory
behind a gated listing, asserting they share one rebuild via the
EnsureVisited singleflight. Add a test that a failed rebuild records the
attempt so an immediate retry is suppressed (per-entry RPC fallback), and
that once the cooldown elapses and the filer recovers a later lookup
rebuilds the cache.
Issue #11262
* mount: wait for pending async flush before rebuilding parent cache
The rebuild lists the parent directory from the filer, so a pending
async flush of the target entry must land first; otherwise the rebuilt
cache captures pre-flush metadata and the cache-hit path returns it
without the wait that guards the filer-fallback path. waitForPendingAsync
Flush is a no-op when no flush is pending, so the common case is unaffected.
Issue #11262
* Add ResolveIdentityClaim helper for OIDC audit identity
ComputeParentUser derives a stable per-identity hash from (sub, iss) for
internal keying, but it is opaque and not human-readable. Audit logs for
STS-assumed OIDC sessions currently surface that opaque value (or the
random session id) as the requester, leaving no authoritative trace of the
federated user.
Add ResolveIdentityClaim next to ComputeParentUser to recover a
human-readable, server-asserted identity attribute from the STS request
context populated at federation time. It walks a priority list
(preferred_username, email, name, sub) so a federated session always
audits against a stable OIDC claim rather than a client-supplied role
session name.
For #11264
* Surface authoritative OIDC identity claim in S3 audit log
For STS-assumed sessions minted from an OIDC web identity, the audit log
requester field is the opaque session subject, which cannot be traced back
to the federated user who performed the operation. The OIDC identity claims
(preferred_username, email, sub) are already carried in the session request
context and reach the auth layer as identity.Claims, but they were never
surfaced to the audit log.
Add a requester_identity field to the S3 access audit log, populated from
the authoritative OIDC identity claim resolved via ResolveIdentityClaim.
The claim is propagated through the shared identity holder (the same
mechanism the requester name and principal ARN already use) so it survives
the request-context copy that hides auth-set values from the outer audit
middleware.
The existing requester field is left unchanged for backward compatibility;
requester_identity is empty for non-federated sessions, where requester
already carries the real username.
For #11264
* Gate OIDC audit identity on federation marker and harden resolver
Address review feedback (Devin Review, Greptile) on the initial
implementation:
- Non-federated STS sessions no longer gain a false requester_identity.
ValidateJWTWithClaims merges the JWT registered sub claim (the opaque
session id) into RequestContext for sessions without an explicit request
context, so the previous ResolveIdentityClaim fallback to sub surfaced
that session id as an authoritative identity. Resolution is now gated on
SessionInfo.ParentUser, which is set only for OIDC-federated sessions in
AssumeRoleWithWebIdentity. The claim is resolved from the original
sessionInfo.RequestContext (not the local claims map, whose sub the bearer
path overwrites with the session subject) so SigV4 and bearer sessions
surface the same identity.
- ResolveIdentityClaim now trims whitespace and treats whitespace-only
claims as absent, so a blank preferred_username no longer masks a usable
email or sub.
The resolved claim is carried on Identity.IdentityClaim (and IAMIdentity for
the bearer path) rather than re-derived in recordIdentityInContext, making
the federation gate explicit at the auth boundary.
For #11264
* Resolve OIDC identity claim for external bearer tokens
The external OIDC bearer path (a raw OIDC JWT presented directly, not via
STS) populates Claims with preferred_username/email/name/sub from the
validated token but did not set IdentityClaim, so requester_identity stayed
blank for that authentication path. Resolve the claim there too — sub is the
real OIDC subject on this path (not an STS session id), so no federation
gate is needed.
Also drop an ineffectual ctx assignment flagged by ineffassign in the audit
test.
For #11264
* refactor: extract update event handling into processUpdateEvent
Pull the OldEntry/NewEntry update branch of the remote sync event
processor into its own function so the rename skip logic can be
exercised by tests with stub clients. No behavior change.
* test: reproduce remote sync rename dropping the new key
A rename under a remote mount arrives as an update whose NewEntry
inherits the source RemoteEntry. shouldSendToRemote returns false
for it, so processUpdateEvent skipped the event without writing the
new key, while the filer had already deleted the old object. The test
runs such an event through processUpdateEvent and expects both a
delete of the old key and a write of the new one. Fails before the
fix. See #11261.
* fix: write the new key when a remote-synced file is renamed
A rename under a remote mount arrives as an update whose NewEntry
inherits the source RemoteEntry, so shouldSendToRemote returns false
(RemoteMtime >= Mtime) and processUpdateEvent skipped the event.
That skip is only valid when the destination key is unchanged; a path
change always needs a write, and the delete-old/write-new handling
below the early return is exactly what a rename needs. Guard the skip
with proto.Equal(oldDest, dest) so a rename falls through to it.
Fixes#11261.
* fix: skip empty upload when renaming a remote-only entry
A remote-only entry (no local chunks or content, data lives only on
the remote object) carries a positive RemoteSize but nothing for
NewFileReader to read. After the previous commit lets a rename fall
through to the delete-old/write-new path, such a rename would upload
EOF and create a zero-byte object at the new key, then stamp it as
synced. Guard the write so a path change on a remote-only entry skips
the upload instead of replacing the file with zero bytes. The filer
has already deleted the old object, so the data is gone regardless;
this avoids leaving a misleading empty object behind.
* fix: propagate old-key delete errors except already-deleted
When deleting the old key on a rename fails for a non-multipart entry,
the error was swallowed and the write proceeded, which could leave both
remote keys. Return the error so MetadataProcessor retries the event.
The filer deletes the source remote object synchronously during the
rename, so the sync delete is redundant and the object may already be
gone. GCS reports that as ErrRemoteObjectNotFound (unlike S3/Azure,
whose deletes are idempotent), so treat it as a successful deletion and
continue to retriedWriteFile rather than pinning the sync offset.
* fix(s3api/iam): fail config snapshot on empty or malformed IAM files
A full IAM reload reads every identity/policy/service-account/group file
from the filer. When an external secrets tool rewrites a file, a reload
that reads it mid-rewrite sees empty or partially-written content. The
identity, policy and service-account loaders silently skipped such files
(``continue``), so the snapshot was missing entries that still existed
on disk. The atomic swap then installed an incomplete identity set while
``isAuthEnabled`` stayed on, denying unrelated clients mid-reload
(#11259).
The group loader and the read-error paths already fail the snapshot in
this situation (a skipped entry reads as deleted). Apply the same
behavior to empty content and unmarshal failures across the identity,
policy, service-account and group loaders, so a transient mid-rewrite
fails the reload (preserving the last known-good state) instead of
silently dropping entries.
* fix(s3api/iam): coalesce burst IAM config reloads through the reload queue
onIamConfigChange did a full synchronous reload for every identity/policy
file change event. When several independently-refreshing credentials
rewrite their files within the same second, that produced a burst of
dozens of back-to-back full reloads, each reading the whole store and
widening the window where a mid-rewrite file is observed (#11259).
Route every IAM config change through the existing coalescing reload
queue (scheduleReload/reloadRetryLoop) instead. A burst of N events now
collapses into a single reload (plus one tail reload for events that
arrived while one was in flight). scheduleReload gains a reason argument
for the existing log line; the reloadRetryLoop already retries failed
reloads, so the per-event failure handoff is no longer needed.
Tests that asserted on the synchronous reload now wire up the queue
(centralized in newTestS3ApiServerWithMemoryIAM) and poll via
waitForIdentity/waitForIdentityGone. Adds TestOnIamConfigChangeCoalescesBurstReloads
showing 50 events coalesce into <=3 reloads.
* fix(s3api/iam): skip non-JSON auxiliary files before failing IAM snapshot
Per review: the multi-file loaders unmarshal every entry in an IAM
directory, so a non-JSON auxiliary file (README, .DS_Store, a migration
backup such as identity.json.old) would hit the new empty/malformed
errors and reject the whole snapshot, blocking all later IAM reloads.
Only *.json files are IAM objects (SeaweedFS writes identities,
policies, service accounts and groups as <name>.json, and other call
sites already gate on the .json suffix). Skip non-.json entries at the
top of each loader loop, before reading content, so auxiliary files are
ignored while empty/malformed .json files still fail the snapshot.
Adds TestLoadConfigurationIgnoresNonJsonAuxiliaryFiles.
* fix(s3api/iam): reject IAM files with empty identifiers and skip aux in listing
Per review:
- ListPolicyNames listed every regular entry in the policies directory as a
policy name, including non-JSON auxiliary files, but GetPolicy cannot
retrieve them. Apply the same .json suffix filter used by the loader so
the list only exposes retrievable policies.
- json.Unmarshal accepts `{}` and unknown fields. The identity and group
loaders merge by the decoded Name (not the file name), so a `{}` file
could install an empty-key record and displace a real one; the
service-account loader accepted an empty Id. Validate Identity.Name,
Group.Name and ServiceAccount.Id (via validateServiceAccountId) after
unmarshal and fail the snapshot on empty identifiers.
Adds TestFilerEtcStoreListPolicyNamesSkipsNonJsonAuxiliary and
empty-identifier regression tests for identity, group and service-account
files.
* rust volume: honour is_last in the tail sender instead of rescanning
volume_tail_sender discarded the is_last flag from
binary_search_by_append_at_ns:
Ok((offset, _is_last)) => {
if offset.is_zero() { Ok(sb_size) } ...
is_last means the caller is already caught up. Go answers that with a
heartbeat and does not scan at all (volume_grpc_tail.go, `if isLastOne`).
Dropping it is expensive rather than untidy, because the branches interact:
when the search reports caught-up it returns Offset::default(), which is
zero, so the start offset falls back to sb_size -- the beginning of the
data -- and scan_raw_needles_from materialises every needle from there to
EOF into a Vec. The timestamp filter discards all of it, the loop sleeps
2s, and it happens again.
A volume being moved is marked read-only before the copy, so it is ALWAYS
caught up during the tail phase. Measured on one volume.move of a 2.15 GB
volume, sampling the source's cgroup anon every 2s against the move's own
phase output:
copying 16 -> 37 MB CopyFile streams correctly, stays bounded
tailing 904 -> 2166 -> 629 -> 2166 -> 342 -> 2173 -> 2179 MB
deleting 46 MB
Six full-volume allocate/free cycles in 35s, peak 2179 MB against a volume
of 2147 MiB. The destination never exceeded 35 MB, so this is entirely
source-side. Under a per-process memory cap it OOM-kills the source
whenever the volume exceeds the cap.
The ordering here is the whole fix and is easy to get wrong: resolve the
start offset and is_last under a brief lock, return the heartbeat
immediately when caught up, and only then reach the scan. An earlier cut
set the flag correctly but placed the early return after the block that
performs the scan -- the heartbeat fired and the destination received
nothing, yet every iteration still read the whole volume and discarded it.
Production showed no improvement (1770 MB across five cycles), which is
what caught it. The binary search is over the .idx and costs nothing; the
scan is the expensive part and must not run speculatively.
Three tests, and the last two matter as much as the first: a fix that
always reported "caught up" would make tailing silently lose needles, a
worse bug than the one being fixed. One asserts is_last for a caller at or
beyond the newest append_at_ns; one asserts NOT is_last for a caller that
is behind, so real tail data is still scanned and shipped; one asserts NOT
is_last when the only newer record is a delete, and that scanning from the
returned offset ships exactly that tombstone.
Left deliberately unfixed, and worth separate changes: the scan still
collects into a Vec rather than streaming through a visitor as Go's
ScanVolumeFileFrom does, and it runs while holding store.read(), the same
lock-across-a-large-read shape as #11235. Both are latent once the rescan
is gone, since remaining scans are bounded by genuinely new data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFr2v4BUqrXdgj4LEUAwVF
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3
* rust volume: resolve and scan the tail under one store guard
The tail sender took store.read() once for the binary search and again
for the scan. A vacuum commit takes the store write lock and swaps
.dat/.idx, so it could land between the two: the offset resolved against
the old files would then be applied to the new ones and start the scan
inside an unrelated record. The code before the is_last fix held a
single guard for both. Restore that, and scan only when the caller is not
caught up, so the caught-up heartbeat still skips the scan and is sent
outside the lock.
Also pin the compacted-volume boundary raised in review. Compaction
writes .idx in needle-id order in both Go and Rust, so the search can
report caught-up while an earlier row is newer; such a caller's since_ns
is the last row's timestamp, so those rows were in the files it copied.
A write made afterwards is appended as the final row, which the search
cannot step past. The new test asserts it still reaches the scan.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3
* rust volume: make the compaction tail test a genuine overwrite
The compaction regression test's second id=1 write reused the first
write's data, so write_needle's dedup short-circuit (is_file_unchanged)
returned without appending or updating append_at_ns. Compaction then
kept key 1's original (older) timestamp, so the test passed without
exercising the overwrite it describes -- key 2 was the final row only
because key 1 was never actually newer.
Give the overwrite distinct data so it appends a new record, and assert
key1_ns > key2_ns up front so a future dedup regression fails the test
instead of silently hollowing it out. Trim the verbose comments on the
tail sender and the binary-search tests to their essentials.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* Refactor parseDurationSeconds into a STSHandlers method
Convert the parseDurationSeconds wrapper from a package-level function
into a method on STSHandlers so it can reach the configured STS service.
No behavior change; the three AssumeRole* handlers now invoke it via
their receiver.
* Respect MaxSessionLength config in STS DurationSeconds validation
parseDurationSeconds validated DurationSeconds against a hardcoded
43200s (12h) ceiling, so raising maxSessionLength in iam.json above
12h had no effect on AssumeRole, AssumeRoleWithWebIdentity, or
AssumeRoleWithLDAPIdentity — requests were rejected at the handler
before reaching the service layer.
Derive the upper bound from the configured STS MaxSessionLength,
falling back to maxDurationSeconds (43200s) when unset. The service
layer (calculateSessionDuration) already caps the issued duration
at MaxSessionLength, so this only relaxes the input-validation gate.
* Add tests for STS DurationSeconds MaxSessionLength bound
Cover the configured MaxSessionLength upper bound, rejection above
it, fallback to the 43200s default when STS config is unset, the
900s minimum, and the empty-parameter nil path.
* Refactor validateSessionDurationSeconds into a STSService method
Convert validateSessionDurationSeconds from a package-level function
into a method on STSService so it can reach the configured STS config.
No behavior change; the three assume-role entry points in the service
(AssumeRoleForPrincipal, validateAssumeRoleWithWebIdentityRequest,
validateAssumeRoleWithCredentialsRequest) now invoke it via their
receiver.
* Respect MaxSessionLength config in STS service DurationSeconds validation
The STS service validateSessionDurationSeconds rejected DurationSeconds
above a hardcoded 43200s (12h) ceiling, so even after the handler
accepted a longer duration it was rejected again in the service layer
for AssumeRoleForPrincipal, AssumeRoleWithWebIdentity, and
AssumeRoleWithCredentials.
Derive the upper bound from the configured MaxSessionLength, falling
back to DefaultMaxSessionLength (43200s) when unset. The issued
duration is still capped at MaxSessionLength by calculateSessionDuration.
* Add tests for STS service DurationSeconds MaxSessionLength bound
Cover the configured MaxSessionLength upper bound, rejection above
it, fallback to the 43200s default when STS config is unset, the
900s minimum, and the nil DurationSeconds path.
* Preserve capping when MaxSessionLength is below the API minimum
Deriving the DurationSeconds upper bound directly from MaxSessionLength
created an empty valid range when MaxSessionLength is configured below
the 900s API minimum, rejecting every explicit DurationSeconds that the
old code silently capped via calculateSessionDuration.
Only apply the configured MaxSessionLength as the upper bound when it is
at least minDurationSeconds; otherwise keep the default bound and let
calculateSessionDuration enforce the shorter configured limit.
* Add tests for sub-minimum MaxSessionLength capping behavior
Verify that a MaxSessionLength below the 900s API minimum keeps the
default upper bound so explicit DurationSeconds within the default
range are still accepted (and later capped by calculateSessionDuration).
* rust volume: stop glibc retaining freed EC buffers as unreturnable heap
A Rust volume server doing EC work accumulates hundreds of MB of resident
anonymous memory that it never gives back, and under a hard cgroup
MemoryMax that ends in an OOM kill while most of the resident set is
free-but-unreturned.
It is not a leak. glibc serves allocations >= M_MMAP_THRESHOLD with mmap
and munmaps them on free, but the threshold is ADAPTIVE: freeing an
mmap'd block raises it toward that block's size, up to 32 MiB. EC
reconstruction and needle reassembly allocate large short-lived buffers,
so the first few train the threshold upward and every later buffer is
carved from the heap instead. Heap pages only return to the OS from the
top of the arena, so they stay resident for the life of the process --
reusable, but anonymous, and anonymous pages cannot be reclaimed under
pressure the way page cache can. The retained footprint is exactly the
headroom a burst of maintenance work needs.
Measured on a 17-node cluster (EC 10+4, --index=redb), one node, two
identical `ec.scrub -mode full` rounds over 10912 EC files each, same
unit restarted with and without a pinned threshold:
baseline round 1 round 2 60s idle
default (adaptive) 10 MB 84 MB 88 MB 88 MB
pinned threshold 10 MB 13 MB 14 MB 14 MB
78 MB retained versus 4 MB for identical work. On heavier mixed scrub
workloads the same effect reached ~600 MB per volume server against a
3 GiB cap, and restarting the process was the only way to release it.
Calling mallopt(M_MMAP_THRESHOLD, ...) sets the threshold and disables
the dynamic adjustment. Pin it to glibc's own default rather than
inventing a value: the goal is to stop the adaptation, not to second-guess
the default. MALLOC_MMAP_THRESHOLD_ still wins if an operator sets it,
glibc-only, and a failed mallopt is logged rather than fatal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFr2v4BUqrXdgj4LEUAwVF
* Address PR review: validate env overrides, honour GLIBC_TUNABLES, fix non-glibc test compile
Three review-bot findings on seaweed-volume/src/malloc_tuning.rs:
1. (CodeRabbit) The test used cfg!(...), which keeps both branches in
compilation. On non-glibc targets DEFAULT_MMAP_THRESHOLD is undefined,
so the test failed to compile. Split into #[cfg]-gated tests so each
branch only references items defined for that target.
2. (Greptile) MALLOC_MMAP_THRESHOLD_ was checked by presence only. An
empty or non-numeric value makes glibc ignore the override while we still
skipped mallopt, leaving the adaptive threshold enabled -- exactly the
behaviour this module exists to prevent. Now we defer only when the value
is non-empty and parses as an integer; otherwise we fall through to
pinning.
3. (Codex) The modern GLIBC_TUNABLES=glibc.malloc.mmap_threshold=... tunable
was missed, so mallopt could overwrite an operator's explicit tunable. Now
we detect that tunable (with the same validation) and defer to it.
The override check moved into the glibc-gated inner function, so off glibc
pin_mmap_threshold() always reports NotApplicable regardless of any
allocator env vars that happen to be set. The startup log for DeferredToEnv
is reworded to cover both override sources. Added tests for the override
parsers and the off-glibc no-op.
* Address round-2 review: match glibc's actual override parsing
Three follow-up review-bot findings after the first round of fixes, all
rooted in our validation not matching how glibc actually parses the
overrides:
1. (Greptile, P1) parse::<i64>() accepted negative values like "-1" and
returned DeferredToEnv, but glibc's threshold is unsigned and rejects
negatives — so we skipped mallopt while glibc also ignored the override,
leaving the adaptive threshold enabled. Now we reject negatives and
zero.
2. (Devin, BUG) glibc parses thresholds as unsigned (strtoul for tunables,
atoi for the legacy var). Values above i64::MAX are valid for glibc but
were rejected by parse::<i64>(), so we pinned 128 KiB over the operator's
explicit setting. Now we parse as u64, accepting the full unsigned range.
3. (CodeRabbit, Major) Two issues in usable_glibc_tunable_threshold:
a. A malformed sibling entry (e.g. glibc.malloc.check=2=2:...) makes
glibc reject the entire GLIBC_TUNABLES string, but our per-entry scan
still returned true for the valid-looking mmap_threshold entry. Now
we validate every entry (exactly one '=') before accepting any.
b. Hex values (0x20000) are accepted by glibc's strtoul but were rejected
by parse::<i64>(). Now parse_strtoul_threshold handles 0x-prefixed hex.
MALLOC_MMAP_THRESHOLD_ stays decimal-only (atoi), matching glibc.
Added regression tests for negatives, zero, >i64::MAX, hex tunables, and
malformed mixed GLIBC_TUNABLES entries. Verified: clippy clean and tests
pass on macOS (non-glibc); glibc-gated code type-checks for
x86_64-unknown-linux-gnu.
* Address round-3 review: match glibc's actual override parsing
Three follow-up review-bot findings (Greptile P1, Devin BUG, CodeRabbit
Major) all on the same issue: the round-2 fix rejected negative and zero
override values, but glibc actually accepts them.
Verified against the glibc source (malloc/malloc.c, malloc/arena.c,
elf/dl-tunables.c, elf/dl-misc.c):
- do_set_mmap_threshold(size_t value) does NO clamping — it just sets
mp_.mmap_threshold = value and mp_.no_dyn_threshold = 1.
- MALLOC_MMAP_THRESHOLD_: glibc calls atoi(value) then mallopt, which
always sets the threshold and disables dynamic adjustment — even for
empty, negative, or non-numeric values (atoi returns 0). So ANY
presence of the variable means the operator's override is in effect.
Reverted to presence-only check for the legacy variable. The round-1
Greptile comment claiming glibc "cannot apply the override" for
empty/malformed values was incorrect.
- GLIBC_TUNABLES: glibc parses values with _dl_strtoul (elf/dl-misc.c),
which accepts decimal, 0x hex, 0 octal, an optional sign (negatives
wrap to unsigned long), and requires the entire value consumed
(tunable_parse_num checks endptr == strval + len). Replaced
parse_strtoul_threshold with dl_strtoul_consumes_all that replicates
_dl_strtoul's parsing and checks full consumption. Now accepts -1
(wraps to SIZE_MAX), 0, 0x20000, 010 (octal), and values above
i64::MAX.
The duplicate-= validation for GLIBC_TUNABLES (from round 1) is kept —
glibc's parse_tunables_string returns -1 if any entry's value contains
a duplicate =, rejecting the entire string.
Added dl_strtoul_consumes_all tests covering decimal, hex, octal,
negative, zero, empty, whitespace, trailing garbage, and sign-only
inputs. Updated usable_glibc_tunable_threshold tests to accept
negative, zero, and empty values. Verified: clippy clean and tests
pass on macOS (non-glibc); glibc-gated code type-checks and clippy
clean for x86_64-unknown-linux-gnu.
* Address round-4 review: add overflow detection, fix sign-only test assertions
Two Greptile P1 findings:
1. Overflowing tunables bypass threshold pinning: dl_strtoul_consumes_all
consumed every digit and returned true for values like
18446744073709551616 (u64::MAX + 1), but glibc's _dl_strtoul stops at
the overflowing digit (sets endptr there, returns UINT64_MAX), so
tunable_parse_num rejects the value (endptr != strval + len). Added
overflow detection matching glibc's cutoff/cutlim logic — on overflow,
the parser stops and returns false.
2. Sign-only parser assertions fail: the test asserted
!dl_strtoul_consumes_all("-") and !dl_strtoul_consumes_all("+"), but
_dl_strtoul skips the sign, finds no digit, sets endptr to the position
after the sign (== end of string), and returns 0. tunable_parse_num
sees endptr == strval + len → true. So glibc accepts sign-only strings
as value 0. Fixed the test assertions to expect true.
Also fixed "0x" with no hex digits: _dl_strtoul parses "0" as octal, then
stops at "x" (not an octal digit), so endptr != end of string → rejected.
The base-detection now requires a hex digit after "0x" before switching
to hex; otherwise "0" is parsed as octal and "x" stops the parser.
Added overflow regression tests: 18446744073709551616 (u64::MAX + 1),
99999999999999999999 (20 nines), 0x10000000000000000 (2^64). Verified:
clippy clean and tests pass on macOS (non-glibc); glibc-gated code
type-checks and clippy clean for x86_64-unknown-linux-gnu.
* Address round-5 review: accept bare 0x prefix, remove unused helper
Two review-bot findings (Devin BUG + CodeRabbit Major) on the same issue:
the round-4 fix required a hex digit after "0x" before switching to hex
base, but glibc's _dl_strtoul unconditionally advances past "0x"/"0X"
when the first char is '0' and the next is 'x'/'X' — even if no hex digit
follows. In that case the digit loop breaks immediately, endptr reaches
the end, and the value is 0. tunable_parse_num accepts it.
Removed the is_digit_in_base lookahead from the base-detection condition
and the now-unused is_digit_in_base helper. Updated the test assertions
for "0x" and "0X" to expect true (accepted as value 0).
The Greptile P1 overflow comment is invalid: glibc's _dl_strtoul rejects
18446744073709551616 (u64::MAX + 1) — on overflow it sets endptr to the
overflowing digit (not end of string) and returns UINT64_MAX, so
tunable_parse_num sees endptr != strval + len and rejects. My
implementation correctly returns false for this value, matching glibc.
Verified: clippy clean and tests pass on macOS (non-glibc); glibc-gated
code type-checks and clippy clean for x86_64-unknown-linux-gnu.
* Address round-6 review: rewrite tunable parser to match glibc exactly
Two Greptile P1 comments (3975151906, 3975151911) both invalid, but
investigation revealed a real bug in the split(':')-based parser:
Bug: usable_glibc_tunable_threshold used split(':') which loses the
distinction between an entry terminated by ':' (glibc skips it) and one
terminated by '\0' with no '=' (glibc rejects the entire string). Examples:
- "glibc.malloc.mmap_threshold=262144:glibc.cpu.x" (no '=' at end):
glibc rejects entire string, old code accepted it.
- "glibc.malloc.mmap_threshold=262144:" (trailing ':'):
glibc rejects entire string, old code accepted it.
Fix: replaced split(':') with a character-by-character parser matching
glibc's parse_tunables_string exactly. The parser tracks position in the
original string and correctly handles all three terminators ('=', ':', '\0')
for both name and value scanning.
Comment 3975151906 (near-maximum values): Invalid. Verified against
_dl_strtoul: for 18446744073709551615 (u64::MAX), cutoff = u64::MAX/10,
cutlim = u64::MAX%10 = 5. After 19 digits result == cutoff. 20th digit 5:
overflow check (digval > cutlim) is 5 > 5 = false → no overflow. glibc
accepts u64::MAX. Added regression test asserting it's accepted.
Comment 3975151911 (later malformed entry): Invalid. Verified against
parse_tunables (elf/dl-tunables.c): when parse_tunables_string returns -1,
parse_tunables prints a warning and returns immediately without applying
ANY tunable — including ones already parsed into the array. Added
regression test for "threshold=262144:check=2=2" (threshold before
malformed sibling) asserting it's rejected.
Added regression tests: u64::MAX accepted, threshold-before-malformed
rejected, no-'=' at end rejected, trailing ':' rejected, leading ':'
accepted. Verified: clippy clean and tests pass on macOS; glibc-gated
code type-checks and clippy clean for x86_64-unknown-linux-gnu.
* Fix CI: correct hex trailing-garbage test assertion
The test asserted !dl_strtoul_consumes_all("0x20000abc"), but in hex
mode a-f are valid digits — "0x20000abc" is a valid hex number
(0x20000abc = 536874044), not trailing garbage. _dl_strtoul consumes
the entire string and tunable_parse_num accepts it. The assertion
failed on Linux CI where the glibc-gated test actually runs.
Replaced with "0x20000g" — 'g' is not a hex digit, so _dl_strtoul
stops at 'g' and tunable_parse_num rejects the value.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(volume): harden volume copy validation
* fix(volume): use stream context for ReadVolumeFileStatus in VolumeCopy
ReadVolumeFileStatus ran on context.Background() while the adjacent
VolumeStatus call used stream.Context(), an inconsistency left over
from the context revert in #11252. Use stream.Context() consistently
so the source status check is cancelled with the VolumeCopy stream.
* fix(volume): reserve destination before deleting existing replica
FindFreeLocation now runs before DeleteVolume so a full target fails
without destroying the existing replica. Previously, when the initial
VolumeStatus check failed (advisory) but ReadVolumeFileStatus
succeeded, the existing replica was deleted before a destination was
reserved, risking data loss if no location had enough free space.
Add a regression test verifying the existing replica survives when
the destination is full and the initial status check fails.
* fix(volume): count replaced replica slot in FindFreeLocation
FindFreeLocation now accepts the volume being replaced so its slot is
treated as available. Without this, a location at its MaxVolumeCount
limit could not replace its sole replica even though deleting it would
free the slot. VolumeCopy passes the volume ID so destination selection
succeeds before the existing replica is deleted.
Add TestVolumeCopyReplacesReplicaAtSlotLimit covering a single-slot
location that must replace its only replica.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: end local-only metadata subscriptions when remote peers appear
SubscribeMetadata delegates to SubscribeLocalMetadata whenever the
MetaAggregator knows no remote peers at stream setup. Peer discovery is
asynchronous with the gRPC server accepting streams: the master announces
filers after Filer.Init, via ListExistingPeerUpdates and OnPeerUpdate.
A subscriber that connects inside that window is pinned to a filer-local
stream for its whole life, silently missing every other filer's writes.
For filer.remote.sync in a multi-filer cluster this means the remote tier
permanently stops receiving writes served by other filers (#11247).
End the delegated local stream when the first remote peer appears, so the
client reconnects into the aggregated stream. The end surfaces as an
error, not a clean EOF: RetryUntil-driven followers (mount meta cache,
s3api IAM) treat a clean end as following finished and stop
reconnecting. The arrival channel is armed under the same lock as the
peer check in RemotePeerArrivedChan, so a peer learned in between sends
the stream straight to the aggregated path instead of parking on a
channel that would never fire.
A standalone filer is unaffected: no peer ever appears, the channel
never fires, and the local stream serves indefinitely.
Fixes#11247
* filer: interrupt disk replay on peer arrival, trim comments
Check upgradeOnRemotePeer inside eachLogEntryFn and chunkDiskPass so a
peer arriving during a backlog replay stops the stream before the
cursor advances past older remote events. Wrap errAggregationUpgrade
with StopReadingError so LoopProcessLogData does not log it. Remove
issue references from comments and trim verbose commentary.
* filer: check upgrade signal between ref batches
Pass upgradeOnRemotePeer to sendRefsBatched so a peer arriving while
refs are shipped to a slow client is detected between batches, not
only after the full batch completes.
* filer: interrupt gap park on peer arrival
Pass upgradeOnRemotePeer through gapPass to parkOnGap so a peer
arriving during a gap park ends the stream immediately instead of
waiting for the retry timer (up to one minute).
---------
Co-authored-by: Tyagiquamar <Tyagiquamar@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: add isActionExplicitlyDeniedByApplicablePolicies helper
Add a helper that reports whether any applicable identity-side policy
(attached IAM policies, enabled-group policies, or the IAM-integration
session policy) explicitly denies an action. It reuses the existing
evaluateAttachedIAMPolicies, resolveS3AuthTarget, buildPrincipalARN, and
isActionExplicitlyDeniedByIAM helpers, and fails closed on evaluation
errors. A nil identity has no identity-side policy plane, so the helper
returns false to keep the bucket policy authoritative for anonymous
access. No behavior change yet; the next commits apply it to the two
bucket-policy Allow short-circuits.
* s3: enforce identity explicit Deny before bucket-policy Allow
authRequestWithAuthType short-circuits on a matching bucket-policy Allow
and skips VerifyActionPermission, so an explicit Deny in an authenticated
identity attached, group, or session policy is bypassed. A non-admin
principal with s3:PutBucketPolicy can install a bucket-policy Allow for
itself and read an object its identity policy explicitly denies.
Before honoring a bucket-policy Allow, check the applicable identity-side
policies for a matching explicit Deny via the new
isActionExplicitlyDeniedByApplicablePolicies helper, and fail closed.
The cross-account behavior is preserved: a bucket Allow still supplies the
Allow an identity policy omits (implicit denial), and a nil identity keeps
the bucket policy authoritative for anonymous access.
Regression tests cover the explicit-Deny override, the implicit-deny
Allow preservation, and the unmatched-key fall-through control.
* s3: enforce identity explicit Deny in secondary object-key auth
authorizeObjectKeyAction authorizes keys the request URL does not name
(CopySource, DeleteObjects body keys, POST Object form keys) and shares
the same bucket-policy Allow short-circuit as the primary path, so an
explicit Deny in the identity, group, or session policy is bypassed the
same way when a bucket policy allows the secondary key.
Apply isActionExplicitlyDeniedByApplicablePolicies before accepting the
bucket-policy Allow, mirroring the primary path. A regression test covers
AuthorizeCopySource for both the explicit-Deny override and the
implicit-deny Allow preservation.
* rust: prevent phantom volumes + validate collection hint in mount_volume_by_id
The collection-hint path (and the find_volume_file_base fallback) called
create_volume on any matching .vif/.idx sidecar. create_volume ->
Volume::new -> load(create_dat_if_missing=true) writes an empty .dat and
registers a phantom normal volume, which can shadow a real EC volume whose
.ecx lives on a sibling disk. This reintroduces the phantom-volume bug the
codebase explicitly guards against in load_existing_volumes.
Apply the same guard load_existing_volumes uses to both paths: only mount
when a real .dat is present or the .vif references a remote-tiered file;
otherwise skip the candidate (no phantom). Also reject path-bearing
collection hints ('/', '\\', '..') so the shortcut cannot route .dat
creation outside the storage directory, falling back to the safe scan.
Adds 3 regression tests; all 336 storage:: tests pass.
Addresses Devin + Greptile review comments on PR #11249.
* rust: address review — .note guard, multi-candidate scan, foo..bar hint
Address the four review comments on #11254:
1. Greptile (P1): contains("..") rejected valid collections like "foo..bar".
Replaced with collection != ".." — volume_file_name joins with "_" so a
".." inside a name is part of the filename, not a parent reference. Only
the exact ".." name is rejected. Added a test that "foo..bar" mounts.
2. Devin #0001 (bug): mount_volume_by_id did not check the .note marker, so
an interrupted VolumeCopy could mount as a live (truncated) volume. Added
a .note check before create_volume in both the collection-hint path and
the fallback — a candidate with .note is skipped (matches
load_existing_volumes). Added a test covering both paths.
3. Devin #0002 + CodeRabbit (major): find_volume_file_base returned only the
first matching candidate, so a lone sidecar on disk 0 hid a real .dat on
disk 1 (the split-disk EC layout the phantom guard protects against).
Added find_volume_file_bases (plural) that collects all candidates; the
fallback now iterates every candidate and mounts the first with a real
.dat or remote .vif. find_volume_file_base delegates to it for
configure_volume. Added a two-disk test: sidecar on disk 0, real .dat on
disk 1 — mount succeeds from disk 1.
All 339 storage:: tests pass (6 mount_volume_by_id tests).
* rust: continue past create_volume failure in mount_volume_by_id
Address Devin review comment on #11254: when create_volume fails on an
earlier candidate (e.g. an unreadable .dat), mount_volume_by_id returned
the error immediately instead of trying later candidates. A valid volume
on another disk remained unmounted.
Both the collection-hint loop and the find_volume_file_bases fallback now
remember the last error and continue scanning. A successful mount returns
immediately; if no candidate succeeds, the last error (or NotFound) is
returned. Matches DiskLocation::open_volumes and Go Store.mountVolume.
Added test_mount_volume_by_id_continues_past_open_failure (chmod 000 .dat
on disk 0, real volume on disk 1, mounts from disk 1).
All 340 storage:: tests pass.
* iceberg: return 401 for invalid or expired Bearer tokens
BUG-0001: when the OAuth JWT expired, Server.Auth fell through to the S3
SigV4 authenticator, which rejects the "Authorization: Bearer" scheme
with NotImplemented — a 501. Iceberg clients (Java OAuth2Manager,
pyiceberg) only refresh tokens on 401, so they retried the dead token
forever: RisingWave sinks stalled and Doris catalog queries failed every
token TTL (1h) until the client process was restarted.
A request carrying a Bearer header is an Iceberg REST client: answer 401
(+ WWW-Authenticate: Bearer, RFC 6750) when the token fails, and only
fall through to the S3 authenticator when no Bearer header is present.
* iceberg: make OAuth token TTL configurable via ICEBERG_OAUTH_TOKEN_EXPIRY
BUG-0001 follow-up: production evidence shows Iceberg Java 1.10.x
clients (RisingWave connector node, Doris FE) never re-fetch tokens on
401 — the sink stalled again on token expiry even with the 501→401 fix,
and no POST /v1/oauth/tokens appeared in server logs across dozens of
retries. 401 is necessary but not sufficient for these clients.
The TTL was hardcoded to 3600 with no knob. Read the expiry (seconds)
from ICEBERG_OAUTH_TOKEN_EXPIRY, defaulting to 3600, so deployments can
issue longer-lived tokens (e.g. 86400) to survive client restart cycles.
* iceberg: support OAuth token exchange (RFC 8693) for client refresh
Decompiling the Iceberg Java 1.10.1 client bundled with Doris FE showed
the missing half of BUG-0001: OAuth2Manager refreshes via token-exchange
(AuthConfig.exchangeEnabled defaults to true — the client_credentials
re-fetch branch only runs with exchange disabled), so a server that only
accepts client_credentials leaves Iceberg clients unable to ever refresh
their token, regardless of 401 correctness.
Accept grant_type=urn:ietf:params:oauth:grant-type:token-exchange on
POST /v1/oauth/tokens: verify the subject_token signature against the
issuing credential, allow exchange within a recovery grace window
(max(2*TTL, 1h), capped 24h) so clients holding tokens that expired
while the grant was unsupported recover without a restart, and mint a
fresh access token with the configured TTL.
* iceberg: harden OAuth token exchange and Bearer matching per review
- match the Bearer scheme case-insensitively (RFC 7235), like
authenticateBearer already does
- accept optional client authentication on the token-exchange grant
(Basic or form credentials, bound to the subject token's client);
expired subject tokens now require it. Iceberg Java's proactive
refresh sends Bearer-only headers, so the grant cannot require it
- reject subject tokens without an exp claim, and re-check the issuer
on the verified claims
- unauthenticated exchange cannot extend the lifetime past the
subject token's own expiry (no chain-refresh from a leaked token)
- return 400 invalid_grant per RFC 6749 §5.2 (was 401)
- include issued_token_type on exchange responses (RFC 8693)
- clamp ICEBERG_OAUTH_TOKEN_EXPIRY to 365d so Duration math cannot
overflow into already-expired tokens
* iceberg: give authenticated token exchanges a fresh full TTL
The remaining-lifetime cap only guards unauthenticated (Bearer-only)
exchanges; an authenticated client renewing a live token must get the
full configured TTL, matching client_credentials.
* iceberg: reject token exchange when no lifetime remains
A Bearer-only exchange with under a second of subject lifetime would
mint a token with expires_in: 0. Reject with invalid_grant instead.
* iceberg: pin near-expiry test token to the next second boundary
jwt/v5 serializes exp at one-second precision, so a 300 ms offset can
round into the current second and route the test through the expired
branch instead of the ttlSeconds<=0 guard. Mint the subject with the
next whole-second expiry: live at exchange time, deterministically
under a second of remaining lifetime.
* iceberg: drop internal ticket reference from comments
* iceberg: clamp oversized OAuth TTLs on 32-bit platforms
strconv.Atoi on an int-sized value fails with ErrRange on 386, so an
oversized ICEBERG_OAUTH_TOKEN_EXPIRY silently fell back to the default
instead of clamping. Parse in 64-bit space and clamp, then narrow.
* iceberg: make OAuth TTL narrowing explicit
* iceberg: disable legacy OAuth in PyIceberg integration tests
* rust volume: move the crate to edition 2024
Edition 2024 turns three things in this crate into hard errors, and changes
drop order in a further 34 places without changing compilation. The compiler
errors are fixed here; the silent changes were audited against
`RUSTFLAGS='-W rust-2024-compatibility' cargo check --all-targets` output
captured before the flip, since edition 2024 stops reporting them.
`std::env::set_var`/`remove_var` are unsafe as of 2024 because they race with
concurrent readers. All six call sites are safe by construction rather than by
assertion, and the SAFETY comments say why: the build script runs
single-threaded before anything else in the process, and every test reaching
the `config.rs` helpers holds `process_state_lock()` for the duration.
The two `ref` bindings in handlers.rs sit in patterns that already borrow
implicitly, so removing the modifier leaves both bindings at `&String`.
On the 34 drop-order sites: no lock guard's scope is extended anywhere, and
`volume.rs` has none. Most are moved-from `Option`/`Result` husks — `if let
Some(v) = map.remove(&k)`, `while let Some(m) = stream.next().await` — where
the value is moved into the binding and the temporary has nothing left to drop;
where closing order actually matters these paths already call `v.close()`,
`ec_vol.destroy()` or `drop(writer)` explicitly. Two sites get strictly better
ordering: the metrics read guard in `run_metrics_push_loop` shrinks to the end
of its initializer block (it never crossed an `.await` either way), and an EC
test now closes the volume's descriptors before the `TempDir` removes the
directory.
No `rust-version` is declared. Edition 2024 needs rustc 1.85, but that is not
the binding constraint — the dependency tree already requires 1.91.1 through
the `aws-sdk-s3`/`aws-smithy-*` family, so `cargo +1.85 check` fails on the
deps regardless. CI builds on `dtolnay/rust-toolchain@stable`.
`vendor/reed-solomon-erasure` is a separate package and keeps edition 2021.
Cargo.lock is unchanged despite edition 2024 implying resolver 3.
Verified: `cargo test` 551 passed / 0 failed, `cargo test
--no-default-features` 550 passed / 0 failed (the two feature sets produce an
identical migration site list), `cargo build --release` clean. No automated
test covers shutdown ordering, so the channel and runtime sites in `main.rs`,
`write_queue.rs` and `grpc_server.rs` were read individually.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC
* rust volume: address edition-2024 review feedback
Three fixes from review of the edition bump.
Serialize the two environment-reading tests. The SAFETY comments on the
`env::set_var`/`remove_var` helpers claim every test touching the environment
holds `process_state_lock()`, but `test_resolve_config_defaults_dir_to_platform_temp_dir`
and `test_resolve_config_index_accepts_redb_and_leveldb_aliases` called
`resolve_config` — which reads HOME/USERPROFILE, SEAWEED_WRITE_QUEUE and the
WEED_* set — without taking it. `set_var` is unsafe precisely because a
concurrent *reader* is UB, not only a concurrent writer, so the comment was
overclaiming. An audit of the module found exactly these two; every other
environment-touching test already held the lock. The race predates edition
2024, which only made the requirement explicit.
Declare `rust-version = "1.91.1"`. The edition needs 1.85, but that was never
the binding constraint: `cargo +1.90 check --all-targets` fails on the
`aws-sdk-s3`/`aws-smithy-*` family, and 1.91.1 checks clean. Declaring the
verified floor turns a wall of per-dependency errors into one clear message.
Cargo.lock is unchanged despite this making the resolver MSRV-aware.
Update the README, which advertised "Rust 1.75+ (2021 edition)". 1.75 was
already stale before this branch — the tree has needed 1.91 for a while.
Verified: `cargo test` 551 passed / 0 failed, `cargo test --no-default-features`
550 passed / 0 failed, `cargo build --release` clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC
* rust volume: state the exact MSRV patch release in the README
The README said "Rust 1.91+", which reads as 1.91.0 and is wrong by one patch
release: `cargo +1.91.0 check --all-targets` fails on the aws-sdk-s3 family,
`cargo +1.91.1` passes. Say 1.91.1+, matching `rust-version` in Cargo.toml, and
call out that the patch component is load-bearing so nobody installs 1.91.0 and
hits the same wall.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC
* rust worker: move the workspace to edition 2024
Moves the seaweed-worker workspace (core, lance, sort) from edition 2021 to
2024, the same migration seaweed-volume just got in this branch.
Edition 2024 turns exactly one thing in this workspace into a hard error. The
baseline came from RUSTFLAGS='-W rust-2024-compatibility' cargo check
--all-targets, run before the flip; unlike seaweed-volume's 34 silent +
8 hard sites, the worker reports only the one hard site and no
tail_expr_drop_order or if_let_rescope sites at all. The worker is a much
smaller crate and none of its expressions hold a guard or temporary whose drop
order the edition changes, so there is nothing to audit on the silent side.
Fixed (1 site):
std::env::set_var is unsafe as of 2024 because it races with concurrent
readers. The single call is in crates/core/build.rs, which sets PROTOC from
protoc_bin_vendored the way seaweed-volume's build script does. A build
script's main runs single-threaded before anything else in the process, so
no other thread can be reading the environment concurrently; the SAFETY comment
says so. There are no config.rs-style test helpers here -- the worker's tests
do not mutate the environment -- so unlike the volume crate there are no
process_state_lock() callers to audit.
No redundant ref bindings to clean up: a grep for ref across the three
crates finds none.
MSRV:
rust-version = "1.94.1", verified rather than inferred. Edition 2024 only
needs 1.85, but the dependency tree needs more: lance's aws feature pulls in
a newer cut of the same aws-sdk-*/aws-smithy-* family that sets
seaweed-volume's 1.91.1 floor, and that newer cut requires 1.94.1.
cargo +1.94.0 check --all-targets fails on that family; cargo +1.94.1
check --all-targets is clean. The worker's floor is therefore higher than
the volume's, and moves with lance and the AWS SDK rather than with the
edition. CI builds on dtolnay/rust-toolchain@stable, so nothing changes
there.
The edition is set once in [workspace.package] and inherited by each member
via edition.workspace = true; rust-version is added the same way. The
workspace keeps its explicit resolver = "2" -- edition 2024 would default to
resolver 3, but the pin is deliberate and Cargo.lock is unchanged by this
commit either way.
The README gains a "Requires Rust 1.94.1+ (2024 edition)" line in its Building
section, matching the one seaweed-volume's README now carries, and calling out
that the patch release is load-bearing (1.94.0 does not build) so nobody
installs 1.94.0 and hits the same wall.
Verification:
* cargo check --all-targets -- clean, zero warnings (default toolchain 1.97)
* cargo +1.94.1 check --all-targets -- clean
* cargo +1.94.0 check --all-targets -- fails on the AWS SDK, as claimed
* cargo test --all-targets -- 40 passed, 0 failed
(core 13, sort 11, lance lib 3, lance bin 2, compaction 6, lifecycle 1,
sort integration 4)
* Cargo.lock unchanged
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
FullPath.DirAndName() and FullPath.Name() used filepath.Split, which is
OS-dependent: on Windows it treats backslash as a path separator,
corrupting filer paths that contain literal backslashes. Filer paths
always use "/" as the separator, so switch to path.Split and path.Join
which only split on "/" regardless of the host OS.
This fixes the backslash case from #11243 where a file saved as
/test/special\reverseslash4.jpg was stored with a corrupted path on
Windows filer builds. The #, ?, and % cases from the same issue are
client-side URL-encoding problems (the server never receives the raw
characters), but once the client properly percent-encodes them the
server now handles the decoded path correctly on all platforms.
* fix: rebase on latest master before pushing star history chart
The daily star history workflow git push was rejected with a
non-fast-forward error because new commits landed on master between
the checkout and the push. Fetch full history (fetch-depth: 0) and
rebase the generated commit on top of the latest remote branch before
pushing so the workflow no longer fails when master has moved.
* fix: retry rebase-and-push to handle concurrent master updates
Address review feedback: a one-shot rebase still races if master advances
between the rebase and the push. Match the bounded retry loop used by
java_release.yml — push first, and on rejection rebase and retry up to
five times before failing.
* fix: serialize runs and ensure every rebase is followed by a push
Address review feedback:
- Devin (line 54-55): the old loop rebased after the 5th failed push but
never pushed the rebased commit. Restructure so each rebase (attempts
2-5) is followed by a push attempt, with a clear 5-attempt cap.
- Greptile: overlapping runs could conflict on the SVG during rebase.
Add a concurrency group (cancel-in-progress: true, matching the repo
convention) so only one chart regeneration runs at a time.
* fix: scope concurrency by ref and guard rebase against transient failures
Address review feedback:
- Devin (line 16): the global star-history concurrency group let a manual
run on another branch cancel an in-flight daily master update. Scope
the group by github.ref so only same-branch runs cancel each other.
- Greptile (line 58): git pull --rebase runs under the fail-fast shell,
so a transient fetch error or conflict aborted the whole step before
remaining attempts ran. Guard the rebase so a failure aborts the
in-progress rebase and continues to the next attempt instead.
* fix(volume): track EC shard read errors and unmount on faulty media
Extract the volume EIO tracker into a reusable IoErrorTracker and add the
same tracking to EcVolume. Sustained EIO on .ecx lookups or .ecd shard
reads now unmounts the EC volume in the heartbeat (without deleting
files) so the master re-replicates from healthy peers, mirroring the
existing volume replica quarantine.
Closes#11227 (EC shard unmount).
* rust(volume): mirror EC shard read error tracking and unmount
Add EIO tracking to the Rust EcVolume mirroring Go: a streak counter
with IO_ERROR_TOLERANCE, a sticky quarantine flag, and unmount (not
file deletion) in the heartbeat so the master re-replicates from
healthy peers.
* feat(metrics): expose storage IO error counter and quarantine gauge
Add a storage_io_error_total counter incremented on every EIO recorded
by the volume or EC shard tracker, and an io_quarantine gauge labelled
by kind (volume/ec_shard) reflecting the count of replicas suppressed
in the heartbeat. Mirrored in Go and Rust.
* feat(healthz): report 503 when local replicas are IO-quarantined
Add Store.HasIoQuarantine (Go) / Store::has_io_quarantine (Rust) and
have /healthz return 503 when any local volume or EC shard is
quarantined due to sustained storage-media EIO, so a load balancer
can drain a server whose underlying media is faulty. Mirrored in Go
and Rust.
* fix(volume): keep quarantined EC volumes in memory and reset EIO on success
Address review feedback: instead of unloading quarantined EC volumes
(which discards the quarantine state healthz needs), keep them in
memory and just skip them from heartbeat reporting, mirroring the
regular volume quarantine. Also clear the EIO streak on successful
.ecx reads in Rust so a transient error does not accumulate, and add
an ec_shard label to the io_quarantine gauge in both Go and Rust.
* fix(volume): exclude quarantined EC shards from heartbeat and add Rust volume tolerance
Address review feedback:
- Filter quarantined EC volumes from CollectErasureCodingHeartbeat
(Go) and collect_ec_shard_delta_messages / collect_live_ec_shards
(Rust) so the master stops advertising faulty shards and
re-replicates from healthy peers.
- Add consecutive EIO count and sticky quarantine to the Rust
regular Volume, mirroring Go IoErrorTracker: a single EIO no
longer deletes the replica; the heartbeat quarantines after the
tolerance threshold and keeps the volume in memory.
- Use the quarantine flag (not last_io_error) in has_io_quarantine
so /healthz reflects sustained, not transient, failures.
* fix(volume): make Rust quarantined volumes read-only and wire recovery
Address Devin review:
- Set no_write_or_delete on Rust volumes when quarantined in the
heartbeat, so cached or direct clients cannot mutate a faulty
replica after the master removes it (mirrors Go).
- Wire reset_io_error_state into Volume::set_writable so an operator
making a volume writable again clears the sticky quarantine and
the volume re-enters heartbeat rotation.
* fix(volume): clear EC quarantine on shard re-mount for operator recovery
Address Greptile review: re-mounting EC shards (Go loadEcShardWithIdxDir
/ Rust mount_ec_shards_with_idx_dir) now calls ResetIoErrorState on the
existing EcVolume, giving operators a documented recovery path that
clears the sticky quarantine and returns the EC volume to heartbeat
rotation. Mirrored in Go and Rust.
* fix(volume): do not clear EC quarantine on routine shard mounts
Address review feedback: clearing the EC IO quarantine on every mount
(including duplicate, retry, sibling-shard, and reconciliation mounts)
is too aggressive and can re-advertise known-bad shards before the
storage media has been validated. Remove the automatic reset from the
mount path; quarantine clears naturally on restart or full unmount
when a fresh EcVolume is created with clean state.
* test(volume): update Rust IO error test for quarantine semantics
The heartbeat now quarantines a volume with sustained EIO (keeps it
mounted, makes it read-only, omits it from heartbeat) instead of
deleting it. Update test_collect_heartbeat_deletes_io_error_volume to
assert the volume stays in the store with no_write_or_delete set, and
update set_last_io_error_for_test to set the consecutive error count
at the tolerance threshold so the test reflects a sustained error.
* fix(volume): reset EIO streak after full write and match Windows media errors
Move the success-side EIO reset from append_needle (after write_all only)
to the end of do_write_request, after flush_dat/flush_idx complete, so a
successful write_all followed by a failed fsync no longer resets the
counter before the EIO is recorded. Repeated fsync EIOs now accumulate
toward the quarantine threshold as intended.
Recognize Windows storage-media failure codes ERROR_CRC (23) and
ERROR_IO_DEVICE (1117) in addition to Unix EIO (errno 5), so quarantined
heartbeat behavior is preserved on Windows. Mirrors the change in both
Go and Rust volume servers.
* fix(volume): preserve checkpoint EIO and clear streak on successful delete
maybe_checkpoint_index now returns whether the checkpoint succeeded;
the success-side EIO reset in do_write_request and do_delete_request
only fires when it did, so a checkpoint media failure is no longer
erased by the unconditional reset that followed it. do_delete_request
also gains the success reset that was lost when append_needle stopped
clearing the streak, so a successful delete still clears an earlier
failure streak.
is_storage_io_error now uses libc::EIO on Unix instead of a hard-coded
5, and the ECX binary-search read path gains a Windows fallback
(seek + read_exact) so the buffer is no longer zeroed on non-Unix
targets.
PR #11236 added a render-time guard that fails the chart when admin.ip
is non-loopback (default 0.0.0.0) and admin auth is not configured,
since weed admin 4.46 refuses to bind a non-loopback address without
authentication. Several pre-existing helm_ci.yml test cases enable
admin.enabled=true as part of "everything on" renders without a
password, so helm template now exits non-zero and the Verify template
rendering step fails.
Add admin.secret.adminPassword to the four render calls that turn on
admin without auth (IAM gRPC opt-in, NetworkPolicy EVERYTHING, egress
without kubeApiServer.cidrs, and the license ALL_ON dict), using the
same key ci/admin-values.yaml already uses.
* volume server: release the store lock before scrubbing EC volumes
`ec.scrub` makes a Rust volume server stop serving for the duration of the
scrub, and then kills its own gRPC connection:
error: rpc error: code = Unavailable desc = keepalive ping failed to
receive ACK within timeout
Measured on a 4.46 cluster (17 Rust volume servers on one host, ~520 volumes
and 53 EC volumes, --index=redb, EC 10+4). It reproduces against a SINGLE
node in 30-70s, in checksum, index and local modes, at -maxParallelization 1.
## Cause
The CHECKSUM arm of scrub_ec_volume reads every byte of every local shard
while holding the caller's store.read() guard:
let store = self.state.store.read().unwrap();
let ecv = store.find_ec_volume(vid)...?;
let (blocks, broken, errs) = ecv.checksum_scrub(); // GBs of I/O, lock held
VolumeServerState::store is a std::sync::RwLock, which is write-preferring.
The periodic heartbeat's collect_heartbeat_with_snapshot takes store.write()
and blocks; once that writer is pending, every later store.read() queues
behind it. Every HTTP handler takes store.read(), so the node serves nothing,
stops heart-beating, and cannot answer the scrub RPC's own keepalive - the
scrub kills the connection it is running on.
The INDEX and LOCAL arms have the same shape, and the node-wide scrub_volume
loop is worse: it held ONE guard across every volume on the node.
## Evidence
offcputime, off-CPU stacks >1s in a 30s window during a scrub:
futex_wait
seaweed_volume::server::heartbeat::collect_heartbeat_with_snapshot
- tokio-rt-worker
27967020 <- 27.97s blocked, of a 30s window
A single HTTP /status request issued 12s into a scrub, with 180s of patience,
was accepted and queued for 120 seconds, then served once the scrub released.
Thread states throughout: 1 D + 48 S. One thread working, 48 idle - not
executor starvation and no thread pileup, which is what a single lock holder
looks like.
Memory was tested and ruled out as the cause: the same scrub was run at
MemoryMax 3G, 8G and unlimited. With no limit there is no reclaim at all,
page cache grows freely to 22 GB, and the node still goes unresponsive at
t+30s. anon stays flat at 48-86 MB in every run.
## Fix
checksum_scrub, scrub_index and scrub_local gain plan types -
EcChecksumScrubPlan, EcIndexScrubPlan and EcLocalScrubPlan - snapshotted from
the volume under a brief guard. The handler builds a plan, drops the guard,
and runs the scan in spawn_blocking, off the async workers, since it is
synchronous CPU + file I/O either way.
A plan captures DESCRIPTORS, not paths. Resolving a path again after the
guard is dropped would let a writer that legitimately unlinks the files - the
heartbeat's delete_expired_ec_volumes, which reaches EcVolume::destroy(), or
volume_ec_shards_delete - surface an intentional removal as "scrub read
error: No such file or directory" and put the volume in broken_volume_ids. A
descriptor outlives the name.
For the shards it duplicates the handle the mounted EcVolumeShard already
holds (try_clone_file), which is what Go does: ChecksumScrub reads through
shard.ReadAt (weed/storage/erasure_coding/ec_volume_scrub.go:71), never
through a path. That also inherits open_volume_file's O_NOATIME and drops a
dead branch - the old code built {base}.ec{id}.v{gen} for a non-zero
generation, a name nothing in this tree writes. dup shares the kernel offset,
so shard reads stay positional; the .ecx gets a fresh open instead, since
check_index_file seeks.
FULL/READS is unchanged here: it already released the guard across the index
walk, and still re-takes it per needle in store_ec::scrub_snapshot_under_lock
for that needle's local shard intervals - short holds, many of them.
scrub_volume now takes the read guard PER VOLUME instead of across the whole
loop, so the heartbeat can land between volumes. Its per-volume work still
runs under the guard; Volume needs an equivalent plan to fix that properly,
left as a follow-up and noted in the code.
## A failed scrub task must not take the whole RPC down
Moving the scans into spawn_blocking changed where a panic lands. It no
longer unwinds inside the handler's own future; it comes back as a JoinError
at the .await, and all four join points sat behind a `?`. So one bad volume
out of six hundred returned Err from the entire handler: the
broken_volume_ids, broken_shard_infos and details already gathered for the
other 599 were dropped, and emit_scrub_metrics - the only writer of
SCRUB_LAST_TIME_SECONDS, SCRUB_VOLUME_FAILURES and SCRUB_SHARD_FAILURES - was
never reached, so the staleness alert kept firing while real corruption went
unreported.
And there is a reachable panic behind it. EcLocalScrubPlan::run() sized its
reassembly buffer with
Vec::with_capacity(get_actual_size(size, version) as usize)
which for any negative size that is not the -1 tombstone skipped above is a
capacity-overflow abort. Mode 3 (LOCAL) is the default of `weed shell
ec.scrub`, and a scrub is what you point at an index you already suspect, so
an arbitrary i32 in a .ecx size field is in-scope input. The buffer is
Rust-only - Go appends to a nil slice and has no capacity hint here. Guard on
`want <= 0` and fall through with an empty buffer: locate_data returns no
intervals for a non-positive size, read stays 0, and the existing
`read != want` error reports the row exactly as Go does.
Each join point now records the failure against its own volume and continues.
A panic is evidence about the volume and counts as broken; a non-panic
JoinError is not - spawn_blocking only reports one when the runtime is going
down, the volume was never scanned, and counting it would put a false
corruption into SCRUB_VOLUME_FAILURES. total_volumes moves before the join in
modes 1, 3 and 4 (2|5 already counted there) so a failed join cannot silently
shrink it. Mode 2|5's verify_ec_shards join is the one that must not
`continue`: the needle walk above has already produced findings for that
volume.
The tombstone guard stays is_tombstone() on purpose. ScrubLocal in
ec_volume_scrub.go:228 skips only IsTombstone(), while the distributed walk
in store_ec.go:516 skips all IsDeleted() - the asymmetry is Go's, and both
Rust walks mirror their own counterpart.
## Both servers: a node-wide scrub skips a volume that vanished mid-run
Releasing the lock makes the volume set legitimately mutable during a scrub,
so a node-wide run can reach a volume that has since been unmounted. That is
not a scrub failure. A node-wide run now logs and skips it; an explicitly
requested volume id still returns NotFound. The Go server is changed the same
way, so both implementations answer the same shell command identically.
mark_broken_volumes_readonly tolerates the same teardown one step later,
instead of throwing away the whole scrub report.
## Test
test_scrub_plans_are_self_contained_and_match_direct_call drops the EcVolume
and runs both plans on another thread, asserting the results match the direct
calls. A plan that borrowed from EcVolume could do neither, so the test stops
compiling if the snapshot regresses to a borrow.
test_scrub_plans_survive_files_removed_after_snapshot unlinks every shard and
the .ecx after the plans are built, then asserts the results still equal the
direct call. Against a path-resolving version it fails with all 14 shards
reported as "No such file or directory".
test_local_scrub_plan_reports_negative_size_ecx_row rewrites a .ecx row's
size to -1000 and runs the local plan on another thread, so the join is the
assertion - that thread is the spawn_blocking whose panic used to fail the
RPC. Without the capacity guard it fails with "capacity overflow"; with it,
the row is reported.
The Go tests cover both halves of the vanished-volume rule for volumes and EC
volumes.
517 lib tests pass, plus 34 across the other targets (`cargo test`).
`go test ./weed/server -run Scrub` passes.
## Known remaining, not fixed here
`ec.scrub -volumeId=N` is still fanned out to every node, and a node that
holds no shard of N returns NotFound, so the shell command errors even when
the nodes that do hold shards scrub cleanly. That is a shell-side fan-out
question rather than a volume-server one, and both servers keep the existing
behaviour for an explicitly requested id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DvHoW85w6SNKNBPvrqLMmK
* scrub: discard checksum block count from total_files; capture .ecx fd for FULL walk
Two review fixes:
1. CHECKSUM arm: plan.run() returns blocks scanned, not a file count.
Go discards it (_, shardInfos, serrs = v.ChecksumScrub()) so TotalFiles
stays a needle/file count. The Rust arm was adding it to total_files,
inflating the count. Discard it to match Go.
2. FULL/READS (scrub_ec_volume_distributed): the needle walk reopened the
.ecx by PATH after the store guard was released, so a concurrent teardown
that unlinks or replaces the .ecx (heartbeat delete_expired_ec_volumes,
volume_ec_shards_delete) could surface an intentional removal as a scrub
error or mix index generations within one scrub. Capture a second .ecx
descriptor under the guard (the index plan handle is consumed by its own
structural walk, and both seek) and read through it instead -- the same
descriptor-outlives-name invariant the checksum plan shard handles use.
* scrub: bind FULL/READS walk to one encode generation
Address Devin review: after capturing the .ecx descriptor under the guard,
scrub_snapshot_under_lock still re-resolves the volume by id per needle, so
a teardown-and-remount of the same vid between two rows would apply the
captured .ecx offsets to a replacement volume's shards -- falsely reporting
corruption.
Capture the volume's encode_ts_ns (encode-run identity) in Phase A and pass
it to scrub_snapshot_under_lock. If the mounted volume's encode_ts_ns no
longer matches, abort the walk like a mid-scan unmount instead of mixing
generations within one scrub.
* scrub: run FULL/READS index scan in the blocking pool
Address CodeRabbit review (5147767192): index_plan.run() reads the whole
.ecx synchronously, so running it on the async executor worker could block
unrelated RPC work handled on the same executor. Move it into spawn_blocking,
matching the treatment the CHECKSUM/LOCAL arms already give their plans. A
join failure (panic/cancellation) is reported as a seed error so the
per-volume findings below are not silently dropped.
* scrub: move ecx walk to blocking pool, classify join errors, guard encode_ts_ns==0
Three CodeRabbit review fixes (5148034447):
1. Move the FULL/READS needle walk (walk_index_file over the captured ecx
descriptor) into spawn_blocking. It reads the full .ecx synchronously and
was still running on the async executor worker, the same blocker the
index_plan.run() fix in the previous commit addressed.
2. Preserve JoinError classification in both spawn_blocking join points in
scrub_ec_volume_distributed. A panic is evidence about the volume and
counts as broken; a cancellation only happens at runtime shutdown, the
volume was never scanned, and returning it as an error would put a false
corruption into broken_volume_ids (the FULL/READS arm marks the volume
broken on any non-empty errs). Panics return an error; cancellations
return clean.
3. Do not treat encode_ts_ns == 0 as a verified generation match. The .vif
assigns 0 when it carries no encode-run identity (legacy/pre-feature
volumes), so 0 == 0 would accept a teardown-and-remount and apply the old
.ecx offsets to the replacement volume's shards. Only enforce the
generation check when the captured identity is non-zero; when it is zero,
fall back to the pre-check behavior (no generation binding) rather than
aborting a scrub that was already running without the guard.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The metadata-subscription readahead channels were sized for a
low-throughput era and now bottleneck replay catch-up:
- ReadPersistedLogBuffer's readaheadSize was 1024 entries: the
background visitor fills the channel, then blocks on the consumer's
gRPC Send, so volume-server I/O for the next log file never overlaps
with delivery of the current one. Each disk pass takes longer, and
the subscribe loop re-lists log files (ListDirectoryEntries on the
filer store) more often to drain the same backlog. Raised to 8192
so the reader stays ahead of the consumer through a full log file's
worth of entries.
- readFilersMerged's logEntryChannelSize was 512 entries per filer
stream: the same serialization on the client side, where weed mount
(chunk mode) reads persisted log chunks directly from volume
servers. A small channel means the producer stalls on the merge
consumer's processEventFn, and the next log file's chunks are never
fetched ahead. Raised to 4096 so volume I/O overlaps with event
delivery.
The wider buffers keep the producer goroutines reading through a full
log file while the consumer is still processing the previous one,
turning serial read→process→read into pipelined read∥process. This
cuts the per-pass wall time that drives filer store listings and
volume-server round-trips, reducing filer workload under backlog
catch-up (e.g. CSI deployments where ~200 mounts reconnect on filer
restart).
* admin: allow insecurely binding to any IP if -allowInsecureNoAuth is set
* admin: rename -allowInsecureNoAuth to -allowInsecureBind
The new flag name is shorter and clearer: it describes what is being
allowed (an insecure bind to a non-loopback address) without the
redundant "NoAuth" suffix.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: pass -ip to admin so StatefulSet becomes Ready on 4.46
Since weed admin 4.46 changed its default listen address from all
interfaces to loopback (127.0.0.1), the admin StatefulSet template never
passed -ip, so the admin server bound to loopback only. The chart's httpGet
readiness/liveness probes dial the pod IP, not loopback, so the probes
never succeeded and the admin StatefulSet stayed 0/1 forever — breaking
upgrades with helm --wait or GitOps controllers.
Add an admin.ip value (default "0.0.0.0", restoring the pre-4.46
behaviour) and render it as -ip. A non-loopback bind requires
authentication, so fail at render time when admin.ip is non-loopback and
neither admin.secret.adminPassword nor admin.secret.existingSecret is
set, instead of letting the pod crash-loop. Document the value and add a
chart-testing CI values file.
Bumps chart to 4.46.1.
Fixes#11234
* helm: address review feedback on admin bind validation
Align the chart's loopback classification with weed admin's isLoopbackIp
(net.ParseIP + IsLoopback): the whole 127.0.0.0/8 range and ::1 are
loopback; localhost and wildcard addresses are non-loopback, matching
the binary. Previously the exact-string check rejected valid loopback
addresses like 127.0.0.2 while permitting localhost (which the binary
treats as non-loopback).
Recognize WEED_ADMIN_PASSWORD supplied via admin.extraEnvironmentVars /
admin.secretExtraEnvironmentVars as authentication, since weed admin
picks it up through viper's AutomaticEnv. Previously such deployments
were wrongly rejected at render time.
Remove [https.admin] mTLS from the validation message and docs: the
chart only generates [grpc.admin] (gRPC mTLS), not [https.admin] (HTTP
mTLS), so mentioning it as an alternative was misleading.
Document that the -ip flag requires SeaweedFS 4.46 or newer, so pinning
admin.imageOverride to an older image is not supported with this chart.
Extracted the loopback and auth checks into reusable helpers
(seaweedfs.admin.isLoopbackIp, seaweedfs.admin.authEnabled) following
the existing seaweedfs.filer.mysqlEnabled pattern.
* helm: tighten loopback classification to reject malformed 127.x addresses
Use regexMatch instead of hasPrefix for the IPv4 loopback check so
malformed values like "127.not-an-ip" are not accepted as loopback
(net.ParseIP returns nil for them, so weed admin treats them as
non-loopback). Also recognize the expanded IPv6 loopback form
"0:0:0:0:0:0:0:1" in addition to "::1", matching net.ParseIP behavior
for the two common representations.
* filer/postgres: create default filemeta table on startup
The postgres filer store hardcoded CreateTableSqlTemplate to empty and
never created the filemeta table, unlike postgres2/mysql2/sqlite which
all create it during Initialize. Users had to create the table manually
or the filer would crash loop with "relation filemeta does not exist".
Read the createTable config option (same as postgres2), default to
DefaultCreateTableQuery when unset, and execute CREATE TABLE IF NOT EXISTS
on the default table after the connection pool is established.
SupportBucketTable stays false so per-bucket table creation remains a
no-op; only the shared filemeta table is created, via a direct ExecContext
since AbstractSqlStore.CreateTable short-circuits without bucket support.
* filer/postgres: accept boolean createTable = true/false
viper reads a TOML boolean as the string "true"/"false" via GetString, so
createTable = true was being used as a SQL template and failed. Add
ResolveCreateTableQuery to normalize the value: true and empty select the
default template, false disables table creation, anything else is a custom
template. Both postgres and postgres2 now use it, and both skip the CREATE
TABLE call when the resolved template is empty.
* scaffold: document createTable option for postgres filer store
Replace the commented-out CREATE TABLE SQL in the [postgres] scaffold with
a createTable config hint, matching the [postgres2] section. Users no longer
need to manually create the filemeta table before starting the filer.
* filer/postgres: make createTable opt-in for postgres, keep postgres2 default
The previous commit defaulted postgres to create the filemeta table even
when createTable was unset, which could break existing deployments whose
DB user lacks CREATE TABLE privileges. ResolveCreateTableQuery now returns
empty for an unset value so postgres only creates the table when
createTable is explicitly true or a custom template — preserving the
prior no-DDL behaviour for existing configurations.
postgres2 keeps its existing always-create default: it defaults an empty
resolved value to DefaultCreateTableQuery, and only skips when
createTable is explicitly false.
* filer/postgres2: simplify createTable handling, document all modes
Drop the false opt-out from postgres2 — it only skipped the default table
while per-bucket CreateTable still ran, leaving restricted DB roles broken
on bucket access. postgres2 now accepts true the same way (defaulting to
DefaultCreateTableQuery) and keeps its existing always-create behaviour
for every other value, matching the original semantics.
The scaffold comment now documents true/false/custom for the postgres
section so users know false (or unset) is the backward-compatible default.
* filer/postgres2: normalize false via ResolveCreateTableQuery
postgres2 only handled "" and "true", leaving createTable = false as the
literal string "false" which CreateTable then executed as invalid SQL.
Route it through ResolveCreateTableQuery (which maps false to empty) and
default the empty result to DefaultCreateTableQuery, so false is treated
the same as unset for the bucket-aware store.
* filer/postgres2: honor createTable = false for default table
postgres2 treated false the same as unset and always created the default
filemeta table, failing startup for restricted DB roles that explicitly
opted out. Track the original false value before ResolveCreateTableQuery
collapses it to empty, and skip the default CreateTable call when set.
Per-bucket table creation is unaffected — it is a runtime requirement of
the bucket-aware store. Users who need to suppress all DDL should use the
postgres (non-bucket) store with createTable unset.
* filer/postgres2: disable bucket tables when createTable = false
Setting SupportBucketTable = false when createTable is explicitly false
makes AbstractSqlStore.CreateTable a no-op (it already returns nil when
SupportBucketTable is false), so neither the default filemeta table nor
per-bucket tables are created. The template stays empty and no DDL runs,
honouring the opt-out for restricted DB roles. All data routes to the
pre-provisioned filemeta table, matching the postgres (non-bucket) store.
* filer: suppress DDL without disabling bucket routing
Setting SupportBucketTable = false when createTable = false also disabled
per-bucket routing, hiding objects in pre-provisioned per-bucket tables.
Keep SupportBucketTable true and instead skip the CREATE TABLE execution
when the resolved template is empty.
GetSqlCreateTable now returns empty for both postgres and mysql SQL
generators when CreateTableSqlTemplate is empty, and
AbstractSqlStore.CreateTable skips the ExecContext call when the SQL is
empty. This preserves bucket routing while suppressing all DDL for users
who explicitly set createTable = false and pre-provision their tables.
* filer: add SkipDDL to suppress CREATE and DROP without disabling routing
createTable = false with SupportBucketTable = true preserved bucket
routing but deleteTable still executed DROP TABLE on bucket deletion,
dropping externally managed tables. CanDropWholeBucket also returned
true, so the S3 layer tried whole-table drops instead of row-by-row
deletes.
Add a SkipDDL flag to AbstractSqlStore, independent of SupportBucketTable.
CreateTable and deleteTable both skip when SkipDDL is set, and
CanDropWholeBucket returns false so bucket deletion falls back to
row-by-row metadata deletes. postgres2 sets SkipDDL when createTable is
explicitly false — bucket routing is preserved, no DDL runs.
* filer: fall back to row-by-row delete when CanDropWholeBucket is false
DeleteFolderChildren took the whole-table drop path whenever the path was
a bucket root, even when SkipDDL made deleteTable a no-op. The no-op
returned nil, the caller returned early, and rows inserted after the
recursive enumeration survived the bucket deletion.
Gate the whole-table drop on CanDropWholeBucket so the row-by-row
DeleteFolderChildren SQL runs when SkipDDL is set, removing all metadata
without issuing DROP TABLE.
* admin: add IsPolicyAttached helper to detect user/group attachments
Introduces AdminServer.IsPolicyAttached, which lists the users and groups
that still have a managed policy attached, reusing the existing credential
manager ListUsers / ListAttachedUserPolicies / ListGroups / GetGroup
methods. This is the building block for rejecting policy deletion while a
policy is still referenced, so deleted policy names stop lingering in a
user attached policy names list (issue #11225).
* admin: reject IAM policy deletion while still attached
Guards AdminServer.DeletePolicy with the new IsPolicyAttached check and
returns the typed ErrPolicyStillAttached error when the policy is still
referenced by a user or group. This matches AWS IAM and the existing IAM
API handler behavior, fixing the stale reference where a deleted policy
name kept showing up in a user attached policy names list (issue #11225).
* admin: return 409 Conflict when deleting an attached IAM policy
The admin UI DeletePolicy handler now maps ErrPolicyStillAttached to HTTP
409 Conflict instead of 500, so the dashboard can surface the attachment
conflict to the user rather than reporting a generic server error.
* admin: skip vanished groups when checking policy attachments
IsPolicyAttached now treats a group that disappears between ListGroups
and GetGroup (credential.ErrGroupNotFound) as no longer attached instead
of failing the whole deletion with HTTP 500, matching the IAM API handler
which skips vanished groups.
* test: assert policy state after deletion paths
Strengthen GetPolicy assertions in the policy deletion tests to check the
returned policy is non-nil after a rejected deletion and nil after a
successful one, not just that no lookup error occurred (GetPolicy returns
nil, nil when a policy is absent).
* fix(s3api): use direct peer IP for aws:SourceIp in bucket policy engine
extractSourceIP in the bucket-policy engine trusted X-Forwarded-For and
X-Real-Ip whenever the TCP peer looked private (loopback/RFC1918/link-local),
with no configurable trusted-proxy allowlist. In containerized deployments
the gateway peer is almost always private, so any caller reaching it directly
or from a co-located workload could spoof aws:SourceIp and bypass
IpAddress/NotIpAddress bucket-policy restrictions.
Always return the direct peer address (r.RemoteAddr), matching AWS S3
semantics. Remove the now-unused isPrivateIP helper and header-trust branch.
Update TestExtractConditionValuesFromRequestSourceIPPrecedence to assert the
peer IP is used regardless of forwarding headers, and add regression tests
TestExtractSourceIP_IgnoresForwardedHeaders and
TestExtractSourceIP_EnforcesIPRestrictionPolicy.
* fix(s3api): use direct peer IP for aws:SourceIp in IAM role/session policies
The IAM middleware's extractSourceIP trusted X-Forwarded-For and X-Real-IP
whenever the TCP peer looked private (loopback/RFC1918/link-local), with no
configurable trusted-proxy allowlist. In containerized deployments the gateway
peer is almost always private, so any caller reaching it directly or from a
co-located workload could spoof aws:SourceIp and bypass IpAddress/NotIpAddress
conditions on role and session policies (IsPrincipalActionExplicitlyDenied).
Always return the direct peer address (r.RemoteAddr), matching AWS S3
semantics. Remove the now-unused isPrivateIP helper, privateNetworks table,
and its init().
Update TestRequestContextExtraction and TestIPBasedPolicyEnforcement to assert
the peer IP is enforced regardless of forwarding headers, and add regression
test TestUserInlinePolicySourceIpCondition_IgnoresForwardedHeaders.
* iam: expose tri-state result from attached policy evaluation
evaluateIAMPolicies returned a bool that collapsed explicit Deny and
no-match into a single false, so the authorization path could not tell
"policies forbid this" from "policies say nothing". Introduce
evaluateAttachedIAMPolicies returning Allow/Deny/NoMatch and keep
evaluateIAMPolicies as a bool projection for existing callers. This is
preparation for unioning native permissions with attached policies while
preserving deny-always-wins.
* iam: preserve native Admin when IAM policies are attached
Attaching an IAM policy routed authorization exclusively to the attached
policies, dropping the identity native permissions. A user with native
Admin lost all access after attaching a non-granting policy, and stayed
locked out if that policy was deleted without being detached first
(#11226).
Treat a native bare Admin grant as a permission floor that survives
attached policies: when the attached policies do not explicitly allow,
fall back to isAdmin() on the attached-policy path, and on the IAM
integration path allow unless an attached policy explicitly denies.
Explicit Deny still wins on both paths.
Only bare Admin is consulted because inline policies flatten lossily into
Actions (dropping conditions), so scoped actions are not unambiguously
native and must keep flowing through the policy engine.
* iam: regression tests for native Admin surviving attached policies
Reproduces issue #11226:
- TestNativeAdminSurvivesAttachedPolicy: a user with native Admin keeps
Write access after attaching a policy that does not grant it.
- TestNativeAdminSurvivesDeletedPolicy: the same user keeps Write access
after the attached policy is deleted without being detached.
- TestAttachedPolicyExplicitDenyOverridesNativeAdmin: an explicit Deny in
an attached policy still constrains a native admin (deny-always-wins).
* iam: apply native Admin floor before IAM principal validation
The native Admin floor in authorizeWithIAM ran after the auth-path
switch, which denies when no session principal or PrincipalArn is
present. An Admin identity without a PrincipalArn (no session token)
was therefore denied before the floor executed. Move the floor ahead of
the switch and derive the principal for its explicit-deny check with
buildPrincipalARN, which already handles identities without a
PrincipalArn. Adds a regression case for an Admin identity with an
empty PrincipalArn.
Addresses CodeRabbit review feedback on PR #11232.
* fix(filer): bound retained reader cache buffers by bytes
* test(filer): keep in-flight downloads during cache trimming
* feat(mem): expose pooled allocation capacity for byte reservations
* fix(mount): share a configurable reader buffer budget across files
* fix(filer): release failed prefetch slots and memory reservations
* feat(mount): expose a soft Go runtime memory limit
* docs(filer): restore shared-download rationale in startCaching
The one-line comment replacing the original context.Background() explanation was too thin for readChunkAt to cross-reference shared resource semantics. Restore a concise note on why request cancellation must not abort a download shared by concurrent readers.
* test(filer): loosen reader cache test deadlines to 5s
Three tests used 1-second deadlines that can flake on CI under load:
TestReaderCacheBudgetInFlight, TestReaderCacheEvictionDoesNotHoldCacheLock,
and TestReaderCacheFailedPrefetchReleasesBudget. Increase to 5 seconds.
* test(filer): cover re-read after reader cache eviction
Add TestReaderCacheReReadAfterEviction: reads chunk 'a', reads chunk 'b'
(evicting 'a' via budget pressure), then re-reads 'a' and asserts a
fresh download returns correct data. Verifies the core correctness
property that eviction never exposes missing or stale data to readers.
Allow normal post-tag proxy propagation before dispatching downstream releases, while preserving the check that prevents them from pinning the previous commit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-07 21:38:49 -07:00
690 changed files with 59692 additions and 13055 deletions
@@ -17,10 +17,11 @@ SeaweedFS is a simple and highly scalable distributed file system. There are two
1. to store billions of files!
2. to serve the files fast!
One `weed` binary serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently.
One `weed` binary serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Both read and write operations have O(1) complexity and can run at the full speed supported by the underlying hardware.
- [Download Binaries for different platforms](https://github.com/seaweedfs/seaweedfs/releases/latest)
- [SeaweedFS White Paper](https://github.com/seaweedfs/seaweedfs/wiki/SeaweedFS_Architecture.pdf) and introduction slides: [2025.5](https://docs.google.com/presentation/d/1tdkp45J01oRV68dIm4yoTXKJDof-EhainlA0LMXexQE/edit?usp=sharing), [2021.5](https://docs.google.com/presentation/d/1DcxKWlINc-HNCjhYeERkpGXXm6nTCES8mi2W5G0Z4Ts/edit?usp=sharing), [2019.3](https://www.slideshare.net/chrislusf/seaweedfs-introduction)
@@ -400,8 +401,6 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
The text of this page is available for modification and reuse under the terms of the Creative Commons Attribution-Sharealike 3.0 Unported License and the GNU Free Documentation License (unversioned, with no invariant sections, front-cover texts, or back-cover texts).
| `recursive` | delete a non-empty directory tree | false; when the filer runs with `filer.options.recursive_delete=true`, deletes are recursive unless `recursive=false` |
| `ignoreRecursiveError` | keep deleting remaining entries after an error | false |
| `skipChunkDeletion` | remove only the metadata, keep volume data | false |
### Tagging
Tags are carried as `Seaweed-`-prefixed request headers, not query
parameters; `?tagging` selects the tagging handler and `?tagging=K1,K2`
lists the keys to remove. Header names are canonicalized on write
(`Seaweed-k1` is stored as `Seaweed-K1`), and the delete list is matched
case-sensitively against the stored names.
```bash
curl -X PUT -H "Seaweed-k1: v1" -H "Seaweed-k2: v2""http://localhost:8888/dir/file.jpg?tagging"
{{- if and (not (include "seaweedfs.admin.isLoopbackIp" $adminIp)) (ne $adminAuthEnabled "true") }}
{{- fail (printf "admin.ip is set to %q (non-loopback) but admin authentication is not configured. Since `weed admin` 4.46 refuses to bind a non-loopback address without authentication, the admin container would exit on startup. Set admin.secret.adminPassword or admin.secret.existingSecret, or supply WEED_ADMIN_PASSWORD via admin.extraEnvironmentVars / admin.secretExtraEnvironmentVars, or set admin.ip to a loopback address such as 127.0.0.1 (note: a loopback bind makes the chart's httpGet readiness/liveness probes fail), or set admin.allowInsecureBind to true to opt out via -allowInsecureBind (INSECURE: exposes the admin API unauthenticated on the network)." $adminIp) -}}
{{- end }}
apiVersion:apps/v1
kind:StatefulSet
metadata:
@@ -162,6 +167,7 @@ spec:
-v={{ .Values.global.seaweedfs.loggingLevel }} \
{{- end }}
admin \
-ip={{ .Values.admin.ip | default "0.0.0.0" }} \
-port={{ .Values.admin.port }} \
-port.grpc={{ .Values.admin.grpcPort }} \
{{- if or (eq .Values.admin.data.type "hostPath") (eq .Values.admin.data.type "persistentVolumeClaim") (eq .Values.admin.data.type "emptyDir") (eq .Values.admin.data.type "existingClaim") }}
@@ -170,14 +176,17 @@ spec:
-dataDir={{ .Values.admin.dataDir }} \
{{- end }}
{{- if .Values.admin.masters }}
-masters={{ .Values.admin.masters }}{{- if or $urlPrefix .Values.admin.extraArgs }} \{{ end }}
-masters={{ .Values.admin.masters }} \
{{- else if .Values.global.seaweedfs.masterServer }}
-masters={{ .Values.global.seaweedfs.masterServer }}{{- if or $urlPrefix .Values.admin.extraArgs }} \{{ end }}
-masters={{ range $index := until (.Values.master.replicas | int) }}${SEAWEEDFS_FULLNAME}-master-{{ $index }}.${SEAWEEDFS_FULLNAME}-master.{{ $.Release.Namespace }}:{{ $.Values.master.port }}{{ if lt $index (sub ($.Values.master.replicas | int) 1) }},{{ end }}{{ end }}{{- if or $urlPrefix .Values.admin.extraArgs }} \{{ end }}
-masters={{ range $index := until (.Values.master.replicas | int) }}${SEAWEEDFS_FULLNAME}-master-{{ $index }}.${SEAWEEDFS_FULLNAME}-master.{{ $.Release.Namespace }}:{{ $.Values.master.port }}{{ if lt $index (sub ($.Values.master.replicas | int) 1) }},{{ end }}{{ end }} \
{{- end }}
{{- if $urlPrefix }}
-urlPrefix={{ $urlPrefix }}{{- if .Values.admin.extraArgs }} \{{ end }}
-urlPrefix={{ $urlPrefix }} \
{{- end }}
{{- if .Values.admin.allowInsecureBind }}
-allowInsecureBind \
{{- end }}
{{- range $index, $arg := .Values.admin.extraArgs }}
{{ $arg }}{{- if lt $index (sub (len $.Values.admin.extraArgs) 1) }} \{{ end }}
Opts::new("SeaweedFS_build_info","A metric with a constant '1' value labeled by version, commit, sizelimit, goos, and goarch from which SeaweedFS was built."),
Opts::new("SeaweedFS_build_info","A metric with a constant '1' value labeled by version, commit, sizelimit, goos, and goarch from which SeaweedFS was built."),
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Chris LuandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
github-actions[bot]
Javier GarciaandGitHub
ihnokimandGitHub
Alex HuandGitHub
Sascha KühndelandGitHub
Ishwar KanseandGitHub
Tobias GurtzickandGitHub
hsdfatandGitHub
Ilia DemianenkoandGitHub
David ChristopherandGitHub
Feng ShaoandGitHub
