* volume: repair an unindexed .dat tail at load instead of loading read-only
A needle write appends the needle to the .dat, then its entry to the .idx,
then answers the client. A crash between the two appends leaves a .dat that
ends past the last indexed needle; on the next start CheckVolumeDataIntegrity
reports "actual N bytes expected M bytes" and the volume loads read-only, and
a complete needle in that tail cannot be read because nothing indexes it.
After a process crash those bytes belong to a write that was never answered;
after a power loss a complete needle can survive in the .dat while its .idx
entry did not, although the client was answered. So complete needles are
re-indexed rather than cut.
CheckVolumeDataIntegrity now repairs the tail before the existing check,
which then runs on the repaired files as before. The repair:
- acts only on a local, version 3 volume within MaxPossibleVolumeSize, with a
whole number of .idx entries, that is about to load writable (neither
noWriteOrDelete nor noWriteCanDelete is set);
- requires the needle at the highest indexed offset to match its entry (id,
size, whole record, CRC);
- walks from the end of that needle to the real end of the file and gives
every complete needle (header, body, checksum, timestamp and padding inside
the file, CRC valid, append timestamp later than the needle before it) the
.idx entry the volume server would have written, in .dat order; a size 0
record, which is how a delete is appended, is indexed as a tombstone;
- cuts only an incomplete last record: fewer bytes than a header, a record
running past the end of the file, or a last record whose CRC fails;
- changes nothing otherwise (a damaged record with more data after it, a
header with id 0 or a negative size, append timestamps going backwards), so
the volume loads read-only exactly as before.
The .idx entries are written and synced before the .dat is truncated and
synced, so a crash during the repair leaves the needles indexed and the
remainder still past them; the next load cuts it. One warning per repaired
volume names the needles re-indexed and the bytes cut.
Tests: volume_checking_tail_repair_test.go reloads volumes through the normal
load path. Repaired: one and two complete unindexed needles, an unindexed
delete, a partial header, a partial body, a complete needle followed by a
partial one, a last needle with a bad CRC, a partial needle after an indexed
tombstone; each reloads clean afterwards. Left read-only and unchanged: the
last indexed needle with a wrong id or a bad CRC, a bad CRC followed by more
data, zero bytes past the last needle, an index past the data, a volume
persisted read-only, a partial .idx entry, and a clean volume.
TestCheckVolumeDataIntegrityVerifiesDeletionTail now also checks that on a
writable volume its 64 zero bytes are reported and kept.
Fixes#11864
* volume: fail the integrity check when a tail repair cannot complete
A repair that wrote the .idx or cut the .dat and then failed to sync only
logged the error; the check that followed saw the repaired files and the
volume loaded writable on a repair that may not be on disk. Any repair
error other than an unexplained tail now fails the check, so the volume
loads read-only, and the next load repairs again.
A failed .idx append could leave part of an entry behind, which made every
later load reject the .idx as a partial entry. The appended bytes are
truncated away again before the error is returned.
* volume: do not cut a tail record that could not be read
The walk cut a last record whose read failed for any reason, so an I/O
error (EIO) on a complete record erased it. Only a record whose bytes are
damaged (needle.ErrorCorrupted: a CRC mismatch or a body that does not
parse) is cut now; any other read error is returned with both files left
as they are, and the volume loads read-only.
* volume: treat a sized tail record with no data as damaged
A needle with no data is written with size 0. A tail record whose header
gives a body but whose data size field reads 0 parsed as an empty needle,
and ReadBytes checks the CRC only when there is data, so the repair indexed
it with its positive size and reads returned an empty body. Such a record
is now damaged like one with a bad CRC: cut when it is the last record,
otherwise the tail is left alone and the volume loads read-only.
* volume: keep an unindexed empty write as an empty needle
A delete and a write with no data both append a size 0 record, and the
repair indexed every size 0 record in the tail as a tombstone, so an
acknowledged empty write whose .idx entry was lost read back as deleted.
A delete is only appended for a needle that is live, so a size 0 record
of a needle that is absent or deleted (in the .idx and in the tail before
it) is an empty write and is re-indexed with size 0. One of a live needle
stays a tombstone, the reading weed fix and the backup's index
regeneration already give it. The .idx is scanned only when the tail holds
a size 0 record.
* volume: cut torn-tail test records inside their body, not at +40
The partial-record cases cut the .dat 40 bytes into a record. A needle
with up to 7 bytes of data is a 40-byte record, so newRandomNeedle's
random length (1..1023) left it whole about once in 150 runs per case:
the repair then correctly re-indexed it and cut nothing, and the test
failed on the .dat size. The cut now lands halfway through the record's
body, which is always past its header and short of its end.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* volume: run the durable sync outside the reader-visible lock
Reads of a volume waited on that volume's fsync because the write lock
was held across Sync()/sync_all. A new per-volume lock (Go
dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat
mutation and lifecycle op across append+publish+sync+rollback, while the
fsync itself runs with dataFileAccessLock/the io write guard dropped —
so readers proceed during a sync, and a failed sync's rollback still
cannot clobber another write or a compaction swap.
Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk)
and Rust (write_needles_grouped/write_synced_run).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: publish after sync; fix Close/CommitCompact lock order
Address review on #11863:
- Close took dataFileSyncLock before waiting on isCompactionInProgress
while CommitCompact claims the flag then takes the lock — a deadlock.
Close now claims the flag before taking the locks (flag -> sync -> access).
- Go now publishes needle-map entries only after the .dat sync, matching
Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes,
publishWriteRequest/publishDeleteRequest index them after Sync. A failed
sync rolls back bytes no reader ever resolved — the streamed-read
offset-invalidation window is gone. processBatch splits runs at repeated
needle ids (Rust parity) so dedup/cookie checks stay sequential; the old
snapshot/restore rollback machinery is removed.
- Rust commit_compact (store-lock caller) takes data_file_sync_lock via
try_lock -> WouldBlock before the write lease; commit_compact_held and
the tier-down _held paths document/require the caller-held sync lock,
and the two vol.instance().write_lock() grpc sites take it first. The
swap can no longer land between a durable write's append and publish.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: restore write watermarks when an inline fsync fails
The syncWrite rollback truncated the .dat but left lastAppendAtNs,
lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and
lastModifiedTsSeconds describing the rejected write; recovery and
compaction could read a watermark for a record that no longer exists.
Snapshot and restore them on failure, as processBatchRun already does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 12:58:02 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: mask anonymous misses behind AccessDenied unless the bucket is listable
Deferred anonymous GET/HEAD authorization only runs once an entry is
selected, so misses returned 404 while existing keys returned 403 — an
unauthenticated existence oracle for bucket names and object keys.
Route not-found responses in Get/HeadObjectHandler through a check that
only discloses 404 when anonymous listing is allowed (policy, identity or
a public-read bucket); otherwise the miss is indistinguishable from a
denied read, matching AWS and 4.48 behavior.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: mask missing-directory misses for anonymous reads too
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: check a fixed request shape when masking anonymous misses
The miss check passed the caller's request through, so query parameters
could retarget the resolved action (?policy) or satisfy an s3:prefix
condition on an unrelated subtree. Each check now runs on a request with a
fixed shape: a ListBucket scoped to the probed key and a plain read of the
key, so the miss only shows when the caller could learn the same fact
directly.
* s3: mask conditional-header misses for anonymous reads
An If-Match conditional on a missing key wrote NoSuchKey directly from
processConditionalHeaders, before the handlers reached the masked miss
path, so anonymous callers could still distinguish a missing key from a
denied read.
* s3: keep the versionId when masking anonymous misses
The read leg must evaluate the action the caller's request resolves to:
a versioned miss is s3:GetObjectVersion, and dropping versionId would let a
plain GetObject grant disclose versions the caller may not read.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The auto-select branches of ReceiveFile (is_ec_volume) and
VolumeEcShardsCopy passed the build-default DATA_SHARDS_COUNT into
find_ec_shard_target_location, so its free-slot maths scored every
volume as 10+4 even when the volume's .vif declares a custom ratio
(e.g. 6+2). A wider-ratio volume's shards could be steered onto a disk
too full to hold them all, and a narrower one would be refused slots
it actually has.
Add Store::lookup_ec_data_shards(collection, vid), which answers from
the mounted EC volume first and then from the .vif on whichever disk
holds it (via read_ec_shard_config_across_dirs, so split -dir/-dir.idx
and rebuild-staging layouts are covered), falling back to the default
when no metadata exists yet.
Mirrors the enterprise fix (seaweed-mono c22fa8a15).
* master: detect and quarantine duplicate volume ids across layouts (#11763)
* shell: volume.check.duplicate_ids audit command (#11763)
* master: persist duplicate-vid quarantine across heartbeats (#11763)
* master: quarantine all copies of a duplicated volume id
Near-simultaneous registrations let each side quarantine only the other,
so arrival order decided which copy stayed writable. A confirmed
cross-layout collision means either copy could be the impostor, so flag
every colliding copy read-only regardless of registration order.
* topology: return a copy from VolumeLayout.Lookup
Callers iterating the returned slice raced with registrations and
removals replacing the location list in place.
* master: release duplicate-vid quarantine when the conflict resolves
A cross-node volume.move briefly shows the same id on two nodes under
different disk layouts; permanent quarantine would pin a valid target
read-only. Keep the first-reported state per quarantined copy and lift
the marks once the conflicting copy unregisters, so only a persistent
collision stays read-only.
* shell: include ec shards in volume.check.duplicate_ids
Ec-only volumes share the global id space. Compare shard collections
against regular ones while allowing shards and regular copies of the
same encoded volume.
* master: keep quarantine release current and cover all removal paths
Track the latest server-reported read-only flag while quarantined so a
release restores the server s true state, and run the release check on
collection deletion and data-node removal too, not only explicit volume
unregistration.
* shell: do not flag ec-only volumes as duplicates
An ec-only volume has no regular copy, so its shard collections had
nothing to compare against and always read as foreign.
* master: quarantine every conflicting layout at the layout level (#11763)
- collect all conflicting layouts, not just the first, so a vid spanning
three layouts does not leave one writable
- mark quarantine on the location list only: folding it into the stored
report made the volume digest permanently disagree with the server and
forced a full volume list every heartbeat
- isAllWritable now honors the location read-only mark so a quarantined
copy stays out of writables through resyncs and re-registration
- re-check the conflict before keeping marks, and do not clear a node's
quarantine while it still holds the vid in another layout
- log the full layout keys on both sides of the conflict
* master: push duplicate-vid quarantine to the volume server (#11763)
A quarantined copy stayed writable on the volume server itself for a
client holding an existing file id, since the master-only mark never
reached the server's local write path. Quarantine now also calls
VolumeMarkReadonly (can_delete so expiry still drains), is re-pushed if
the server comes back reporting it writable, and VolumeMarkWritable
runs when the conflict resolves and the server had reported writable.
* master: retry unenforced quarantine marks and restore server writability on release (#11763)
Dial the advertised grpc port, return mark errors, and keep a
quarantine entry until the writable push succeeds so a released
copy does not stay read-only on the server.
* master: treat rp-only layout differences as an in-flight replication change (#11763)
volume.configure.replication rolls through nodes one at a time, so a
volume legitimately reports the old and new replica placement while it
settles. Layouts differing only in rp within the same collection are a
supported transition, not a reused id.
* shell: group ec shards by collection in the duplicate id audit (#11763)
EC shards of one volume legitimately span disk types when
DiskTypePrefer spills, so grouping by disk type reports healthy
volumes as duplicates.
* pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763)
* master: freeze volume growth on topology id mismatch (#11763)
* shell: volume.growth.resume command (#11763)
* master: replicate the growth freeze and gate it on veteran heartbeats (#11763)
* master: block allocation while the topology id is being minted
DeferAllocation only armed after the topology id persisted, so growth
requests arriving during init ran ahead of the warmup window. Gate
NextVolumeId on an init-pending flag spanning the whole mint.
* master: stash reported topology ids until the local id is minted (#11763)
* drain stashed topology id reports when the id arrives outside the mint path
* persist the growth-freeze flag in the mirror file and raft snapshots
* log hashicorp raft marshal and apply errors in applyGrowthFreeze
2026-10-20 07:34:31 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: group-commit fsync uploads on the direct write path
Each fsync=true upload used to run its own .dat/.idx sync when the write
queue was off, which is the default. Route durable uploads through the
per-volume write workers so a batch shares one sync, matching Go's
writeNeedle2: non-fsync writes, and writes while the server is stopping,
stay on the synchronous path. The queue is now always constructed and
SEAWEED_WRITE_QUEUE is gone.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: document post_handler write routing
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: share the UpdateTable authorization gate
Extract the validation, load, and permission check from handleUpdateTable
into authorizeUpdateTable so the table an update was authorized against is
the same snapshot the mutation writes to. Manager gains
AuthorizeUpdateTable, mirroring AuthorizeCreateTable, so callers can ask the
question without updating anything.
* s3tables: share the UpdateView authorization gate
Extract the lookup, policy load, and permission check from handleUpdateView
into authorizeUpdateView, mirroring authorizeUpdateTable, and expose it as
Manager.AuthorizeUpdateView for callers that need the answer without the
update.
* s3api: scope vended table credentials to read-only on request
VendTableCredentials takes a readOnly flag and tablePrefixSessionPolicy
emits only s3:GetObject for the table prefix when it is set, keeping the
conditioned ListBucket and unconditioned GetBucketLocation statements that
reads still need.
* iceberg: vend read-only credentials when the caller may not write
LoadTable and friends vend storage credentials only for the authority the
caller holds: the vend now checks UpdateTable/UpdateView permission and a
caller that may only read the catalog entry gets a read-only session. Paths
that just passed a write gate (stage-create, view commit) keep the full
session.
* lance: vend read-only credentials when the caller may not write
storage_options vends the same read-only session as the Iceberg catalog
when the caller lacks UpdateTable on the table.
* lance: keep write credentials for callers authorized to declare
A Lance commit is a direct S3 write that never revisits the catalog, so a
caller that just passed CreateTable still gets a writable session; the
UpdateTable gate applies to describe, the read path.
* iceberg, lance: log a failed write check instead of only reading it as denied
2026-10-20 07:16:27 -07:00
Chris LuGitHubDevin <devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: list keys in byte order around directory markers
* s3: order deferred dir blocks by slash name and hold quota
* s3: fix deferred dir blocks at page and marker boundaries
Review follow-ups on the marker byte-order change:
- A marker inside an emitted "<name>/" block re-descended and re-listed its
folded contents under delimiter "/" instead of skipping the block.
- A prefix object's "<name>/" common prefix was emitted eagerly at the name
position, outranking siblings like "a.txt" and repeating across pages; the
plain key now emits alone at the name position and the block emits the
common prefix when children exist.
- A recursion starved on quota before listing a deferred subtree returned
without truncation, dropping keys silently; the block now reports a
resumable truncation when the subtree still holds entries.
- Clone prefix-object entries with proto.Clone rather than copying the
protobuf message by value.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: close the hasEntries probe stream on return
* s3: resume delimited listings at the dir block boundary
A deferred directory block emits at its "<name>/" position, but emitDir
left nextMarker at the bare name, which sorts inside the (name, name/)
band and reads as "block pending". Every pass of the listing loop then
re-emitted the same common prefix until maxKeys ran out (999 prefixes,
IsTruncated), and a "<name>/" continuation token trimmed to "<name>" by
adjustMarkerForDelimiter reopened the block and paged forever.
Advance nextMarker to the block boundary the same way the recursive
branch does, and keep a "/" -ending marker intact — it already names the
bound past the block.
---------
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
heartbeatResponse now reads MaxVolumeId and TopologyId off ms.Topo, but
the test built a MasterServer without one and panicked, failing the
package for every PR.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume balance: unify the per-move convergence guard
Evaluate the post-move utilization comparison through serverUtilization so
the guard also applies when servers lack MaxVolumeCount info (raw-count
mode), where it previously did not run at all.
* volume balance: restrict fallback destinations to improving servers
The score-based fallback inside createBalanceTask could redirect a move to
any valid server, including ones as full as the source — between equally
full servers the move only relocates the imbalance and the next detection
cycle reverses it (#11839). Exclude from eligibleTargets any server whose
post-move utilization would exceed the source's, so a fallback destination
satisfies the same convergence guard as the loop-chosen target.
* volume balance: skip a volume when the chosen target violates placement
When the detection loop's target resolves but cannot host the replica (it
already holds one, or is out of the filtered scope), createBalanceTask now
returns nil so the loop advances to the next volume — matching shell
volume.balance, which continues to the next candidate instead of
redirecting the move to a different server.
The score-based fallback is kept for the case where the target cannot be
resolved at all; its allowed set is already limited to servers the move
would improve.
* volume balance: regression test for replica-constrained targets (#11839)
Reproduces the reported loop: the emptiest server holds the second rp=001
replica of the lowest-ID volumes on the fuller servers, so it is never a
legal target for them. Detection must skip those volumes and send the
movable ones to the under-utilized server, and re-running detection after
applying the plan must never move the same volume twice.
2026-10-19 20:52:30 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The master intentionally sheds an assign that triggered volume growth
once its wait window elapses (503 + Retry-After); on a slow CI box
cold-start growth can outlast it, so the single-shot assertion flaked.
Honor the documented retry contract like a real client, within a 60s
deadline.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: strip internal object-lock and version headers on auth entry
putToFiler persists Seaweed-X-Amz-Object-Lock-Mode, -Retention-Until-Date,
-Legal-Hold and -Version-Id request headers into authoritative entry
metadata without a permission check, so an s3:PutObject-only writer could
pin COMPLIANCE retention and legal hold on its own uploads, bypassing the
dedicated-permission gate on the standard x-amz-object-lock-* headers.
The server only sets these headers from validated inputs, so drop them in
authenticateRequestInternal alongside the identity headers already
scrubbed there.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: strip internal SSE headers on auth entry
X-SeaweedFS-SSE-S3-Key-Data, -SSE-S3-Base-IV and -SSE-KMS-Base-IV are read
from the request to drive multipart encryption, so a client could inject
its own key material and IVs. The server sets them itself between handler
stages of a request; drop inbound values in authenticateRequestInternal.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: scrub internal headers before the auth-enabled early return
Auth and AuthPostPolicy skip authenticateRequestInternal when IAM is
disabled, so client-supplied internal headers still reached putToFiler.
Hoist the scrub into scrubInternalHeaders and run it unconditionally in
both wrappers as well as the shared auth entry point.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: test internal header scrub through the auth-disabled wrapper
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 18:35:20 -07:00
Chris LuGitHubDevin <devin-ai-integration[bot]@users.noreply.github.com>
* filer.sync: keep the needle-404 clock from resetting on live volumes
The give-up clock for a source chunk that cannot be produced is kept per
volume so a vacuumed volume's files all inherit one wait. But a needle
404 and a volume lookup failure shared that clock, and sourceServed
cleared it whenever any needle on the same volume read fine, so on a
busy cluster the grace never elapsed for a single deleted needle on a
healthy volume and filer.sync retried it forever (#11841).
Give needle not-found answers their own per-volume clock that serving
siblings does not reset.
* filer.sync: an answered 404 needs no liveness probe before skipping
After the missing-chunk gate gives up, onReplicateChunkError only skips
the entry when a re-read of the most recently served chunk succeeds.
That probe can never have been armed on a fresh sync, and its single
stored file id can itself be vacuumed since — either way the entry stays
pinned and the offset never advances.
A needle 404 was answered by a live volume server, which is itself proof
the source is serving requests, so skip on it directly. Keep the probe
for the lookup-failure flavor, where no server answered and the silence
can still mean a source outage.
* filer.sync: skip a replayed version the source rewrote in place
Supersession was decided by entry mtime alone, so a source entry whose
chunks were swapped without an mtime bump — remote cache eviction and
re-cache, an in-place repair — never read as superseded, and its
vacuumed chunks were retried indefinitely.
When a chunk is missing, compare the replayed chunk's file id against
the live entry's resolved chunk set: absent means the replayed version
is obsolete and its event is skipped on the first failure instead of
after the give-up window. The same live-entry comparison now also
decides the entry-level skip in onCorruptChunk/onReplicateChunkError.
* filer.sync: give each missing needle its own grace clock
A 404 is one needle's answer, but the clock was kept per volume: once the
first needle's wait expired, every other needle on that volume inherited an
already-elapsed clock and could be written off on its first sighting. Track
the wait per file id and clear it only when that needle itself serves again,
so a live volume's healthy needles neither shorten nor restart it.
* filer.sync: keep the probe for filer-proxied 404s
ReadPart proxied through the source filer collapses "no volume locations"
into the same 404 a missing needle returns, so that answer cannot stand in
for the liveness probe the way a direct volume read's can. Only treat 404 as
an answered needle when the sink reads volume servers directly.
* filer.sync: only a not-older live entry can supersede a replay
Chunk-list comparison must respect the same ordering the mtime check does:
a live entry older than the replayed version means the source was rolled
back, so its differing chunks say nothing about whether the replayed data
is still needed. Gate the content check on live mtime >= replayed mtime in
both sourceMovedOn and sourceDroppedChunk.
* filer.sync: compare chunk ids without resolving manifests
resolvedChunkFileIds downloaded manifest files to expand chunk lists, so a
manifest sitting on a dead volume made the supersession check itself fail
and left the offset pinned behind a replica the source cannot read. File
ids are immutable needle identities — the unresolved sets already compare
chunk content exactly, and the check no longer performs I/O.
* filer.sync: judge nested chunks by their top-level chunk
A manifest child is not in the entry's chunk list even when the manifest
still references it, so checking the live entry for the child id reported
it dropped and returned the raw 404 — unwrapped, so the offset still
pinned. Thread the entry-level chunk id down the manifest recursion and
consult it for the dropped-chunk check instead.
* filer.sync: let the manifest-child test exit through the gate
Run the worker with a shortened grace period and assert the returned error
is marked errSourceChunkMissing, instead of racing the worker still running
against a restored RetryWaitTime.
* s3: sort ListMultipartUploads by key and honor key-marker
* s3: skip upload folders without an object key in ListMultipartUploads
* s3: order same-key multipart uploads by initiation time
The upload id suffix is now a time-ordered UUIDv7, so sorting by upload id lists a key's uploads oldest first, like S3. Pagination is unchanged: ordering and the upload-id-marker comparison share the same key.
selectMultipartUploads now builds the whole result, including NextKeyMarker and NextUploadIdMarker, and the pagination test pages with the response markers.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-19 18:10:06 -07:00
Chris LuGitHubDevin <devin-ai-integration[bot]@users.noreply.github.com>
The HEAD on the completed multipart object occasionally gets a 403 from a
signature verification mismatch before the object read settles; retry the
download instead of failing the run.
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
* pb: heartbeat fields for the federated volume-id bound (#11763)
Heartbeat carries max_volume_id_seen and last_seen_topology_id up, and
HeartbeatResponse echoes max_volume_id and topology_id down, so the
allocation bound is replicated to every volume server.
* master: echo the volume-id bound and fold reported bounds in (#11763)
Every heartbeat response carries the leader current max volume id and
topology id; every heartbeat reported max_volume_id_seen is folded into
the topology bound via UpAdjustMaxVolumeId.
* volume: persist a per-dir vid stamp of the highest id ever seen (#11763)
The store keeps max_volume_id_seen + last_seen_topology_id in a
.vid_stamp file per volume dir, fed by heartbeat echoes, AllocateVolume,
and local volume/EC creation, seeded at boot from stamps and local
holdings, and reported in every heartbeat. A rebuilt master recovers
the bound from the first surviving veteran report.
* volume-rust: mirror the vid stamp and heartbeat echo handling (#11763)
Same mechanism as the Go volume server: .vid_stamp per dir, bound folded
in from add_location scans and allocate/mount calls, echoed fields
applied on heartbeat responses, reported on every heartbeat.
* volume: serialize vid stamp writes and fsync the directory (#11763)
* volume-rust: serialize vid stamp writes and fsync the directory (#11763)
* master: echo the allocation bound to connected servers on change
The bound was only sent at connect time or on digest mismatch, so a
healthy stream never learned later increases. If the high-id server then
went offline and the master lost state, survivors restored a stale bound
and the id could be reissued. Resend the heartbeat response whenever the
bound or topology id moved.
* volume: retry failed vid stamp saves on later updates
A transient write failure left the in-memory bound raised but the stamp
stale, and repeat updates at the same bound never retried. Track a dirty
flag so any later update re-attempts the save.
* volume: cover stamp bootstrap for pre-stamp dirs in test
* volume-rust: retry failed vid stamp saves, mirroring Go
* volume-rust: stamp every dir with the combined bound on add_location
A dir whose own stamp sat below the combined bound kept its stale value
after fold; losing the higher-stamped disk then restarted the server
below ids it had already learned. Rewrite all stamps after each location
folds in.
* volume: stamp mounted volume and ec shard ids (#11763)
* volume-rust: stamp mounted volume ids, mirroring Go (#11763)
* volume: retry dirty vid stamp save on unchanged topology id (#11763)
* volume-rust: retry dirty vid stamp save on unchanged topology id (#11763)
* master: record echoed bound from the sent heartbeat response (#11763)
* s3tables: accept .puffin data files for Iceberg v3 deletion vectors
Iceberg v3 writes deletion vectors as Puffin files under a table's data/
directory, so merge-on-read DELETE and UPDATE on v3 tables were refused
with AccessDenied by the table-bucket upload validator.
Fixes#11842
* s3tables: name all accepted data formats in rejection error
The message stopped at .avro even though .lance and now .puffin are
accepted.
* s3tables: cover deletion-vector filenames in data layout test
Uses the names real engines emit: Spark/Iceberg Java
{seq}-{task}-{uuid}-{counter}-deletes.puffin, UUID-suffixed names, and
the same under a partition directory.
* s3api: check the direct peer against the trusted-proxy allowlist
* s3: verify X-Forwarded-Prefix only from trusted proxies
* s3: note that trusted proxies must sanitize forwarded headers
* s3tables: honor a table policy Deny for maintenance operations
authorizeMaintenanceTable evaluated only the bucket policy, so a Deny in the table policy never vetoed PutTableMaintenanceConfiguration, GetTableMaintenanceConfiguration or GetTableMaintenanceJobStatus even though they act on the table ARN. The stored table policy is now checked for a matching Deny first via deniedByPolicy, a veto helper that keeps the admin exemption, and resource and bucket tags are loaded into the policy context so tag-conditioned Denies evaluate. readBucketPolicy is renamed to readPolicy and reused for the table path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: honor the stored policy's Deny on policy management operations
PutTablePolicy, GetTablePolicy and DeleteTablePolicy only evaluated the bucket policy, so a Deny in the table's own policy was ignored and a bucket-level Allow could strip or replace the policy guarding a table. The existing table policy is now loaded and a matching Deny vetoes the call. PutTableBucketPolicy likewise consults the existing bucket policy for a Deny before replacing it. Resource and bucket tags populate the policy context so tag-conditioned Denies evaluate, and policy writes assert the policy authorized against is still in force, returning a conflict when it changed mid-request.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: apply a table policy Deny to tag operations
TagResource, ListTagsForResource and UntagResource evaluated only the bucket policy; a Deny in the table's own policy is now honored when the target ARN is a table. ListTagsForResource and UntagResource also returned early when the resource carried no tags attribute, skipping the permission check entirely; the check now always runs. Policy evaluation uses the canonical resource ARN rebuilt from stored metadata, so a request ARN with a forged account or region cannot dodge the Deny.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: regression tests for resource policy denies
Cover the case where a bucket policy grants an identity s3tables:* while the table policy denies it: the policy, tag and maintenance operations must stay denied so the protection cannot be stripped. Also cover a bucket policy that denies PutTableBucketPolicy under the open fallback, and a tag-conditioned Deny, which vetoes only while the stored tag matches.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: give Volume interior locks mirroring Go's
Volume's mutable state moves behind interior locks matching Go's
per-volume lock domains, so the store can hand out shared handles
instead of requiring exclusive &mut access:
- io: RwLock<VolumeIo> owns the .dat/remote handles, needle map,
append cursor, compaction bookkeeping and write watermarks —
Go's dataFileAccessLock-adjacent mutable set; helpers take the
guard as &mut VolumeIo so nothing re-locks.
- super_block and volume_info each get a RwLock
(superBlockAccessLock, volumeInfoRWLock); dir_idx gets its own
so path building never contends with I/O.
- no_write packs Go's noWriteLock pair into AtomicU8;
last_modified_ts_seconds and compaction_byte_per_second become
atomics.
The data-file write lease (dataFileAccessLock) now covers every
mutating entry that previously relied on borrow exclusivity:
writes, deletes, blob writes, set_writable's flag/index swap,
remote-file updates, local-backend swaps, index relocation,
replica-placement rewrite, put_needle_index, close and destroy.
set_read_only_persist leases only its can_delete idx-attach branch
because mark_io_unavailable reaches it under a held lease.
commit_compact takes try_write_lock — callers hold the store lock,
so blocking would stall every volume behind one slow reader;
WouldBlock surfaces as COMMIT_WAITS_FOR_READS for the caller to
retry.
Lock order is fixed: lease -> io -> volume_info/super_block/dir_idx.
* storage: mount volumes as shared Arc<Volume> handles
DiskLocation's map owns Arc<Volume> so the store can hand a caller a
handle that stays valid across store-guard releases, instead of
borrowing &mut Volume under the store write lock.
- volumes: HashMap<VolumeId, Arc<Volume>>; find_volume returns a
cloned Arc.
- find_volume_mut and volumes_mut are gone — every Volume method is
&self now, so shared handles cover what exclusive borrows did.
- writable_volume_mut -> writable_volume(&self) -> Result<Arc<Volume>>
keeps the same admission checks (exists, disk-space-low).
- write_volume_needle/write_volume_needles/delete_volume_needle are
&self; lifecycle methods that mutate the map or mounted state stay
&mut self.
* server: run needle I/O with the store guard released
The store write lock previously wrapped needle writes and deletes
end-to-end: process_batch held it across every grouped fsync, and the
HTTP/gRPC fallbacks, tail receiver and fetch-and-write paths did the
same. One volume's slow I/O then stalled admission and metadata on
every other volume.
- write_queue resolves each group's handle via writable_volume under
a brief store read, then runs write_needles_grouped after the guard
drops.
- HTTP write/DELETE fallbacks, the gRPC batch-delete EC recheck +
delete, tail receiver and fetch_and_write local writes, blob writes
and the quarantine/sync/set_writable/remote-file paths all run
under store.read() now that every volume method they reach is
&self.
- Lifecycle calls that mutate the volume map or mounted state
(mount/unmount/delete, EC shard ops, compaction commits,
load_new_volumes, heartbeat volume actions) keep store.write().
* test: cross-volume admission proceeds during another volume's write
Regression coverage for the store-lock fix:
- write_queue's test parks a volume-1 write inside its .dat append via
a new needle_write_hook seam (mirroring needle_read_hook), then
proves the store still hands out a lifecycle writer and a volume-2
handle — while volume 1's data-file lease still excludes same-volume
writers.
- tests/store_lock_repro.rs exercises the public API: concurrent
read-guarded writes on two volumes, and a stale Arc<Volume> handle
failing closed after unmount instead of writing into torn-down
files.
* test: drop store write guards that shared handles made read-only
Test setup sites held store.write() purely to reach find_volume_mut;
with Arc<Volume> and &self methods the guard is read-only (clippy
readonly_write_lock). The volume lease still serializes the writes
inside.
* volume: drop a windows-only read_exact_at that shadows the io helper
cargo's windows build failed E0255: the free fn copied with the
interior-lock conversion collided with the unconditional
crate::storage::io::read_exact_at import, which already carries the
platform branches internally.
* server: release the store read guard before needle I/O
The read guard still ran through write_needle_blob_and_index, the tail
receiver write, the fetch-and-write local write, and the HTTP write and
DELETE fallbacks, so a slow append blocked mount/unmount writers (and,
on writer-preferring locks, readers on other volumes). Resolve the
Arc<Volume> under a short guard and write through the handle after it
drops — the same scoping process_batch already used.
* volume: serialize the read-only mark and its .vif save under the lease
set_read_only_persist only leased the idx-attach branch, so concurrent
mark-readonly and mark-writable calls could interleave flag updates and
.vif writes — including two save_vif calls truncating and writing the
file at once. The lease now spans the whole flag change and save, with
a _held variant for mark_io_unavailable, which reaches this from write
paths already holding the lease (Go: markVolumeReadonly runs under
dataFileAccessLock).
* volume,server: span the tier-down transition under one write lease
update_remote_files cleared the remote-only write restriction and
released its lease before open_local_dat_backend attached the local
file, so a write could slip into the gap — pass check_writable, then
fail appending to a missing dat_file. The handler now holds the lease
across the reference trim, .vif save, and backend swap (and the rollback
re-insert), via _held variants; the store read guard is scoped to the
handle lookup. The swap still lands after the .vif trim — the ordering
that keeps a crash from pointing the volume at a deleted remote object.
* server: run a batch's volume groups on separate scoped threads
A single batch can hold writes for several volumes; running the groups
sequentially made every later volume wait behind a stalled append on an
earlier one — the same head-of-line blocking the store guard release
fixed, one level up. Each group now gets a scoped thread, preserving
order within a group while independent volumes progress in parallel.
* storage: close a volume handle a mount slot replaces
With shared handles, inserting a new Volume over an existing map entry
left the old handle alive in callers — two Volume instances could then
write the same files with separate in-memory indexes. set_volume and
create_volume now close() the replaced handle so its writers fail
closed, matching what unload_volume already does.
* test: key the write hook off a reserved needle id
The hook registry is process-global and keyed by needle id, and other
parallel tests write NeedleId(1) — an unrelated write could enter the
barriers and leave the test's real writer parked forever.
* server: wait out readers for vacuum commit without the store guard
commit_compact try-locked the write lease because callers held the store
write lock, returning COMMIT_WAITS_FOR_READS while a download was in
flight — and the master reads that as a failed vacuum, discarding
completed compaction work. The commit RPC now resolves the handle under
a brief store read and blocks on the volume's lease, matching Go's
CommitCompact under dataFileAccessLock: a slow reader stalls only this
volume's vacuum. The try-lock path stays for callers that still hold the
store lock.
* server: scope the remaining store read guards to volume lookup
Same class as the write-path fix: the batch-delete cookie check and
tombstone, the query read, volume_needle_status, and the local chunk
read all kept the read guard through the needle operation, so slow I/O
there could still stall store writers. Each now resolves the Arc<Volume>
under a short guard; the EC rechecks keep their guards.
* seaweed-volume: fail closed on detached volume handles
A handle resolved under a brief store read can outlive its mount: an
unmount+remount between lookup and lease leaves the Arc pointing at a
closed Volume while a new instance owns the files. Mark the handle in
close() and reject it under the write lease in commit_compact_held,
delete_needle, check_writable, set_read_only_persist and
update_remote_files_held; the tier-down handler also checks the flag
before deleting a remote object its replacement still reads.
* seaweed-volume: drop the store guard before mark-readonly waits on the lease
mark_volume_readonly held a store read across set_read_only_persist, which waits out the volume's leased readers; a slow download would stall every lifecycle operation behind the guard. Resolve the handle under a short read, then mark off the guard; a detached handle now maps to not_found.
* seaweed-volume: retry batch deletes that raced an EC mount
The store guard now drops before delete_needle, so ec.encode can mount shards between the pre-delete recheck and the tombstone; the encode then removes the .dat and the needle resurrects from pre-tombstone shards. Re-check has_ec_volume after each successful delete and return the retriable 503 so the filer requeues onto the EC path.
* seaweed-volume: one write-queue worker per volume
A stalled batch kept later batches for other volumes queued behind it: the single worker awaited process_batch before receiving again. Go's volume_write.go keeps one batching worker per volume, so give each volume its own FIFO worker spawned lazily, bounded by a global concurrent grouped-I/O semaphore and a global queued-request bound; workers exit only when their channel closes, and submit refuses unknown volume ids before spawning.
* seaweed-volume: move the queued-write permit inside the request
submit held the global queued-slot permit across the send and the response wait, so a canceled submit freed the slot while the worker still held the needle; repeated cancels could overflow the bound. The request now owns the permit until the worker drops it.
* seaweed-volume: retire write workers whose volume left the store
worker_for kept a sender per volume forever, so servers that mount and retire volumes accumulated idle workers. A worker now polls while idle and retires once its volume leaves the store: close the channel so racing senders fail fast, drain accepted writes, then release the map slot — a replacement worker can never start while this one still holds queued writes, so same-volume FIFO survives retirement.
* design: durable volume-id allocation boundary (#11763)
* master: mirror max volume id to a file outside raft (#11763)
The allocation bound lived only in raft state, so a master that lost it
rebuilt the bound from whatever volume servers were connected and could
reissue ids held by offline servers.
NextVolumeId now persists the bound to <mdir>/max_volume_id before the
raft apply, and both raft implementations persist on apply and recovery
so follower mirrors stay current. The file also carries the topology id,
which survives raft state cleanup the same way the snapshot recovery did.
* master: serialize and fully sync the max volume id mirror (#11763)
* master: rewrite the mirror when the topology id changes at the same bound
A same-bound raft apply carrying a new topology id was skipped by the
lastPersistedMaxVolumeId check, leaving the mirrored topology id empty
until the next allocation. Track the mirrored topology id too and
rewrite when either value differs.
* drop the design doc from the PR stack
The reverse index walk dedupes seen NeedleIds with a BoomFilters bloom
filter. Its default kernel is FNV-1, whose final step only XORs the last
input byte into the hash's low bits, so keys sharing the high 7 bytes of
a big-endian NeedleId land on correlated probe positions and false
positives arrive in bursts of hundreds instead of the ~0.001 target
rate. Each false positive hides a live-latest key and inflates the
rebuilt deletion counters, which is what made
TestFastLoadingNeedleMapMetrics flake past its +256 bound in CI
(observed DeletedCount drift of 267).
Swap the kernel for xxhash, which fully diffuses the key bytes; the
observed false-positive count in 300 local runs of the test drops to
max 2 (avg 0.23).
* ec: compare live idx NeedleId sets before encode
Refuse encode when replicas' live .idx NeedleId sets differ. Walk only
live rows (skip deleted and Size<=0) and compare NeedleId, ignoring Size
so MIME/flags drift does not abort a matching id set. Mark the volume
read-only for encode and reject writes after that.
* fix(ec): compare all replicas then pick capacity source
Log matching live idx sets with "replicas are consistent" for the
shell CI check. After readonly, compare every replica's live NeedleId
set, then require the encode source to have FreeVolumeCount>=2.
Empty live-idx vs non-empty refuses encode instead of skipping
superblock stubs. Drop the FileCount rebind after the idx gate.
* volume: mark readonly under the write lock so checked appends finish first
* ec: fix readonly deadlock and restore worker-host source preference
MarkVolumeReadonly held dataFileAccessLock while calling
reopenIdxForWrite, which re-acquires the same mutex and deadlocks
TestMarkVolumeReadonlyCanDelete. Split out reopenIdxForWriteLocked for
callers already holding the lock, matching the existing Locked pattern.
The worker task lost its worker-host source preference when the idx gate
replaced the replica union. Late-bind the source to a replica on the
worker host using only the checked location set, with no FileCount
rebinding. Drop the now-unused helpers, trim comments added by the
earlier commits, and rename tests whose names still claimed convergence
while asserting refusal.
---------
Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
A restarted goraft master wipes its raft state and only learns the
TopologyId after the leader re-admits it and replays the entry. That
trails HTTP readiness by several seconds, so the one-shot check in
assertTopologyIdConsistent raced the propagation (CI run 39199033687:
TopologyId applied on the follower ~2s after the test gave up).
* admin: default to port 12535, keep 23646 as a best-effort compat listener
The default admin HTTP port 23646 derives a worker gRPC port of 33646
(port + GrpcPortOffset), inside the Linux default ephemeral range
(32768+). One of our own startup dials can claim 33646 before the admin
binds it, killing startup. This is the same failure #10209 fixed for
the test port allocator, still reachable via the mini default (#10838)
and via standalone `weed admin` defaults.
Move the default admin port to 12535 (gRPC 22535) everywhere, keeping
the http+10000 convention so worker -admin derivation, the Rust worker
and host:http.grpc addressing keep working:
- weed admin -port, weed mini -admin.port
- weed worker -admin, the plugin worker fallback, Rust worker and
weed-lance-worker defaults
- help text, README, helm values, docker compose, tests
So old workers, bookmarks and scripts keep working, admin also serves
the same handler and the worker gRPC server on the previous default
(-port.legacy / -admin.port.legacy, default 23646, 0 disables). The
legacy gRPC port binds best-effort: it lives in the ephemeral range, so
a failed bind only warns instead of reintroducing the startup failure.
weed mini reserves it early like the primary port and keeps it out of
its own port relocation. grpc.Server tracks every listener handed to
Serve, so the compat listener stops with the primary.
Also warn when an explicitly configured admin port puts the derived
gRPC port at or above the ephemeral floor, the other fix suggested in
the issue.
Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc
off the +10000 convention breaks external workers that derive the admin
gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins
-admin.port on their side; this makes that pin unnecessary.
Fixes#10838
* admin: let services win the legacy ports, close compat listener on failure
Review findings on the compat listeners:
- Do not put the legacy ports in mini's reservedPorts: an explicit
-s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected
or stolen by the early reservation. Mini now checks the finalized port
set and skips only the conflicting compat listener, so HTTP and gRPC
compat degrade independently and the service keeps its port.
- Also skip the legacy HTTP listener when a mini service owns the port:
a wildcard-bound service can still leave loopback bindable, which would
let the compat listener shadow it for loopback clients.
- Close legacyServer's raw listener in the deferred cleanup: Serve may
not have registered it yet when startAdminServer returns early.
- Helm: add admin.legacyPort (default 23646) and expose the legacy
http/grpc ports through the admin Service, container ports and
NetworkPolicy so older workers resolved via the Service keep working.
* helm: only pass -port.legacy when it differs from the binary default
ct install deploys the released image with the new chart, and 4.48 does
not know -port.legacy, so the admin pod crash-loops. Omitting the flag at
the default value is behavior-identical on new binaries (their default
is also 23646) and keeps older releases installable; legacyPort: 0 or a
custom port still renders the flag.
* webdav: delete a folder with its contents
* webdav: guard the recursive delete against stale and confined paths
- propagate stat lookup errors instead of returning an error-bearing
FileInfo: OpenFile and DELETE would otherwise take the recursive
removeAll on a path whose type was never confirmed, wiping a
directory on a transient filer error
- refuse wrappedFs.Rename from or to the served root, matching the
RemoveAll guard and webdav.Dir
- normalize trailing slashes in removeAll so the delete names the
entry itself, and refuse the root for every caller
* webdav: keep file-only deletes shallow and refuse ancestor destinations
- removeAll used by OpenFile and failed-upload cleanup now deletes
non-recursively, so a folder swapped in after the stat check is not
emptied by a pending file delete
- MOVE and COPY whose destination contains the source are refused
before the handler removes the destination, which would otherwise
take the source down with it
* webdav: also refuse a destination inside the source
a MOVE or COPY into its own subtree recurses into the destination it
just created; refuse it with the same guard instead of burning the
recursion budget for a 500
* webdav: allow a shallow copy into the source subtree
a COPY with Depth: 0 creates only the destination collection and never
reads children, so it is safe inside the source; only deep copies and
moves are refused there
* webdav: refuse a copy/move whose source is unreadable
* webdav: cover the missing-source refusal in the guard test
* webdav: 500 on a source lookup failure, 404 only when absent
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Brings in the raft snapshot-recovery fixes: a recovered follower returns
to Follower so it can campaign when the leader goes away (masters on
ephemeral -mdir no longer wedge leaderless), and currentTerm stays
monotonic and persisted through recovery.
* filer: free shared chunks when a folder delete drops the last hard link
doBatchDeleteFolderMetaAndData skipped chunk collection for every hard-linked child, relying on maybeDeleteHardLinks to retire the shared inode record. When the delete removes the last link the record is gone and nobody else will ever free those chunks. After the link callback runs, check whether the record survived; if it did not, queue the entry's chunks for deletion.
* filer: free orphaned hard-link chunks in the replayed delete sweep
The subtree sweep on a replayed directory delete skipped every hard-linked child, so a link that was the inode's last owner leaked its shared chunks. Check the inode record after the store delete; when it is gone, queue the child's chunks like any other swept entry.
* filer: mirror remote-object cleanup in the replayed delete sweep
Children a lagging peer alone still holds under a replayed directory delete were dropped with plain store deletes: their remote objects leaked and lazy remote fetches could resurrect them. The sweep now records event-stamped tombstones and deletes swept non-directory children through maybeDeleteFromRemoteIfUnchanged, which carries the same stat guard the write-back daemon's processDeleteEvent applies plus a modification-time check — a delayed replay cannot erase an object another writer replaced, even a same-size one. A failed remote delete fails the replay so the event is retried instead of orphaning the object. The directory's own tombstone comes from the event pipeline at the event timestamp and covers the subtree. Metadata-only deletes and cross-cluster events keep the origin's semantics.
* filerstore: return real errors from KvGet on MongoDB, ArangoDB, Elastic
These stores mapped every lookup failure to ErrKvNotFound, so a database error looked identical to a missing key: hard-link records could appear gone while links still owned their chunks, and path timestamp reads could treat committed state as absent. Return the underlying error and keep ErrKvNotFound for genuinely missing keys.
* filer: bind remote-object deletes to the observed object version
A guarded stat-then-delete still lets a replacement landing between the two requests go down with the deleted object. Add an optional conditional-delete operation to remote storage clients — S3 If-Match on the live ETag, GCS generation match, Azure IfMatch — and use it everywhere a delete is guarded by a stored stamp: the write-back daemon's delete, rename-source, and pending-rename paths, and the replayed-delete sweep. Backends without it keep the guarded stat-then-delete.
* filer: restamp the remote entry after a metadata-only remote update
Changing Content-Encoding rewrites the object in place on S3 and advances its LastModified, while the recorded stamp keeps the old mtime. A version-checked delete would then mistake the object for a replacement and keep it forever. Re-stat after UpdateFileMetadata so the saved stamp describes the live object.
* filer: mirror the remote-delete ownership checks in the replayed sweep
The sweep deleted swept children's remote objects on the listing's snapshot
alone. Two gaps: a remote-only entry that took the path since — one a commit
record may not cover, such as a lazy fetch materialized from the remote —
reads the object as its only copy, and an entry removed while its rename
copy was pending leaves the object at the source key orphaned. The per-child
delete now runs under the child's entry lock with the commit-time check
repeated, keeps a remote-only entry that holds the path, and releases the
pending rename's source under the same ownership and stamp conditions the
write-back daemon's releasePendingSource applies.
* remote.sync: restamp only the object the metadata update touched
The live stat after UpdateFileMetadata was saved unconditionally, so a
replacement written between the update and the stat stamped the entry with
an object it never described — and a later delete would then pass the
freshness guard on the replacement's data. The restamp now requires the live
object to still match the stored stamp on size and ETag; a different object
keeps the old stamp, which the delete guard reads as replaced and keeps.
* filer: skip unbudgeted reader-cache test on 32-bit
TestReaderCacheUnboundedWithoutBudget proves the reader cache has no
implicit memory budget by keeping 96 concurrent 4 MiB downloads in flight
(>256 MiB). On linux/386 the weed/filer test binary already runs near the
4 GiB address-space limit, so the burst reliably crashes it with
"out of memory" mid-suite.
* filer: bail out of a swept child lock instead of waiting on it
Replay already holds the event's path locks when the sweep reaches a
descendant. A local rename can hold that descendant while waiting on one
of the replayed paths, so blocking on the child lock deadlocks both. The
sweep now tries the lock and reports ErrLockContended — transient, so the
replication retry replays the event once the local write settles.
* remote: report the stamp a metadata update produced
A metadata write can rewrite the object it touches — S3 changes
Content-Encoding through an in-place copy, which can mint a new ETag —
so restamping from a fresh stat can reject the very object the update
produced, or describe a replacement that landed in between.
UpdateFileMetadata now returns the RemoteEntry of the object it wrote
(S3 from the copy result, GCS from the update attributes, Azure from a
post-update stat); nil means the object was untouched and callers keep
the stored stamp.
* remote: restamp Azure from the metadata write's own etag
* iam: scope CreateServiceAccount's ParentUser by policy resource
AuthorizeIamAction evaluated the iam:CreateServiceAccount grant against a
hardcoded arn:aws:iam:::* resource, so the policy's Resource never had a
say; #11218 then bound the target to the caller outright. Evaluate
self-target actions against the target identity's ARN instead: a wildcard
resource keeps the deliberate any-target grant, a scoped resource confines
issuance to the named parent identity, anything else is denied.
buildResourceARN gains the ARN pass-through buildS3ResourceArn already has,
so iam: ARNs reach the policy engine intact on the attached-policies route.
Closes#11735
* ci: retrigger after unrelated weed/storage metrics flake
* iam: fold IAM-ARN matching into awsIAMMatch, drop debug test
- remove zz_debug_test.go (local debugging leftover)
- fold awsIAMMatchResource into awsIAMMatch: the arn:aws:iam:: value
prefix selects case-sensitive matching, so no separate name is needed
- trim comments
---------
Co-authored-by: batonogov <batonogov@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: resume into the directory a listing marker names
* s3: keep delimited marker resumes exclusive
* s3: descend a marker-named directory without repeating its key
* s3: emit marker-named dir objects but keep internal folders hidden
* webdav: create a folder under its own name
* webdav: reuse NormalizePath for the MKCOL entry split
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The worker fix reads the source's TTL destroy time through
ReadVolumeFileStatus, which returns the in-memory volume_info. Go's
SaveVolumeInfo stamps volumeInfo.ExpireAtSec before writing; Rust's
save_vif wrote it only to the serialized file, so a Rust source kept
reporting the stale creation-time value and the EC .vif would carry a
wrong expiry. Stamp the live volume_info first, mirroring Go and
save_volume_info.
* sts: recognize issuer-scoped parent user ids
ComputeParentUser output has a fixed shape (base64url SHA-256); let
callers distinguish it from bare subject strings.
* s3tables: keep legacy owner re-anchoring off local and scoped account ids
Re-anchoring a stored bare subject under the single trusted issuer must
only apply to strings that could actually be a legacy OIDC subject. A
stored owner naming the reserved or a configured local account, or an
already issuer-scoped id, can otherwise be claimed by any caller whose
token asserts that string as sub.
* s3tables: re-anchor legacy policy principals in map and array forms
A stored bare subject can appear as {"AWS": sub} or inside a principal
list, not only as a top-level string; re-anchoring only the string form
left legacy Deny statements unable to match the caller.
* s3tables: keep built-in legacy exclusions independent of the account checker
A missing local-account predicate must not disable re-anchoring entirely:
the reserved accounts, ARN-shaped values, and scoped-id shapes are never
OIDC subjects regardless, while legacy Deny statements for plain subjects
still need to resolve so a denied caller cannot fall through to an
identity grant.
* s3tables: exercise legacy Deny against an owner grant
The previous case denied the caller regardless of whether the statement
principal matched; make the caller the owner so a missed match shows.
Cover the array Deny form alongside the AWS map.
* s3api: recheck condition under lock when routed filer ignores condition_alt_key
A filer that predates condition_alt_key silently drops the field and
evaluates a suspended-versioning condition against the empty null slot
alone, so a matching If-Match still gets a false 412. Recheck under the
object lock instead of trusting that response: the lock path resolves the
same entry the read path serves. Genuine precondition failures still
return 412 from the recheck.
* s3api: cover suspended If-Match against a filer ignoring condition_alt_key
* s3api: resend against the resolved entry instead of the lock-path recheck
Rechecking under the gateway object lock does not serialize against
routed writers, which commit under the filer's path lock; a concurrent
routed PUT could land between the recheck and the create and be
overwritten with a stale precondition. Resend the same transaction with
condition_key set to the resolved version file so the condition and the
write stay under the same filer lock. When the resolved entry is the
null slot, a delete marker, or unresolvable, the resend targets the
object path and fails closed.
github-actions[bot]
