Ec-only volumes share the global id space. Compare shard collections
against regular ones while allowing shards and regular copies of the
same encoded volume.
A cross-node volume.move briefly shows the same id on two nodes under
different disk layouts; permanent quarantine would pin a valid target
read-only. Keep the first-reported state per quarantined copy and lift
the marks once the conflicting copy unregisters, so only a persistent
collision stays read-only.
DeferAllocation only armed after the topology id persisted, so growth
requests arriving during init ran ahead of the warmup window. Gate
NextVolumeId on an init-pending flag spanning the whole mint.
A transient write failure left the in-memory bound raised but the stamp
stale, and repeat updates at the same bound never retried. Track a dirty
flag so any later update re-attempts the save.
The bound was only sent at connect time or on digest mismatch, so a
healthy stream never learned later increases. If the high-id server then
went offline and the master lost state, survivors restored a stale bound
and the id could be reissued. Resend the heartbeat response whenever the
bound or topology id moved.
Near-simultaneous registrations let each side quarantine only the other,
so arrival order decided which copy stayed writable. A confirmed
cross-layout collision means either copy could be the impostor, so flag
every colliding copy read-only regardless of registration order.
A same-bound raft apply carrying a new topology id was skipped by the
lastPersistedMaxVolumeId check, leaving the mirrored topology id empty
until the next allocation. Track the mirrored topology id too and
rewrite when either value differs.
Same mechanism as the Go volume server: .vid_stamp per dir, bound folded
in from add_location scans and allocate/mount calls, echoed fields
applied on heartbeat responses, reported on every heartbeat.
The store keeps max_volume_id_seen + last_seen_topology_id in a
.vid_stamp file per volume dir, fed by heartbeat echoes, AllocateVolume,
and local volume/EC creation, seeded at boot from stamps and local
holdings, and reported in every heartbeat. A rebuilt master recovers
the bound from the first surviving veteran report.
Every heartbeat response carries the leader current max volume id and
topology id; every heartbeat reported max_volume_id_seen is folded into
the topology bound via UpAdjustMaxVolumeId.
Heartbeat carries max_volume_id_seen and last_seen_topology_id up, and
HeartbeatResponse echoes max_volume_id and topology_id down, so the
allocation bound is replicated to every volume server.
The allocation bound lived only in raft state, so a master that lost it
rebuilt the bound from whatever volume servers were connected and could
reissue ids held by offline servers.
NextVolumeId now persists the bound to <mdir>/max_volume_id before the
raft apply, and both raft implementations persist on apply and recovery
so follower mirrors stay current. The file also carries the topology id,
which survives raft state cleanup the same way the snapshot recovery did.
* webdav: delete a folder with its contents
* webdav: guard the recursive delete against stale and confined paths
- propagate stat lookup errors instead of returning an error-bearing
FileInfo: OpenFile and DELETE would otherwise take the recursive
removeAll on a path whose type was never confirmed, wiping a
directory on a transient filer error
- refuse wrappedFs.Rename from or to the served root, matching the
RemoveAll guard and webdav.Dir
- normalize trailing slashes in removeAll so the delete names the
entry itself, and refuse the root for every caller
* webdav: keep file-only deletes shallow and refuse ancestor destinations
- removeAll used by OpenFile and failed-upload cleanup now deletes
non-recursively, so a folder swapped in after the stat check is not
emptied by a pending file delete
- MOVE and COPY whose destination contains the source are refused
before the handler removes the destination, which would otherwise
take the source down with it
* webdav: also refuse a destination inside the source
a MOVE or COPY into its own subtree recurses into the destination it
just created; refuse it with the same guard instead of burning the
recursion budget for a 500
* webdav: allow a shallow copy into the source subtree
a COPY with Depth: 0 creates only the destination collection and never
reads children, so it is safe inside the source; only deep copies and
moves are refused there
* webdav: refuse a copy/move whose source is unreadable
* webdav: cover the missing-source refusal in the guard test
* webdav: 500 on a source lookup failure, 404 only when absent
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Brings in the raft snapshot-recovery fixes: a recovered follower returns
to Follower so it can campaign when the leader goes away (masters on
ephemeral -mdir no longer wedge leaderless), and currentTerm stays
monotonic and persisted through recovery.
* filer: free shared chunks when a folder delete drops the last hard link
doBatchDeleteFolderMetaAndData skipped chunk collection for every hard-linked child, relying on maybeDeleteHardLinks to retire the shared inode record. When the delete removes the last link the record is gone and nobody else will ever free those chunks. After the link callback runs, check whether the record survived; if it did not, queue the entry's chunks for deletion.
* filer: free orphaned hard-link chunks in the replayed delete sweep
The subtree sweep on a replayed directory delete skipped every hard-linked child, so a link that was the inode's last owner leaked its shared chunks. Check the inode record after the store delete; when it is gone, queue the child's chunks like any other swept entry.
* filer: mirror remote-object cleanup in the replayed delete sweep
Children a lagging peer alone still holds under a replayed directory delete were dropped with plain store deletes: their remote objects leaked and lazy remote fetches could resurrect them. The sweep now records event-stamped tombstones and deletes swept non-directory children through maybeDeleteFromRemoteIfUnchanged, which carries the same stat guard the write-back daemon's processDeleteEvent applies plus a modification-time check — a delayed replay cannot erase an object another writer replaced, even a same-size one. A failed remote delete fails the replay so the event is retried instead of orphaning the object. The directory's own tombstone comes from the event pipeline at the event timestamp and covers the subtree. Metadata-only deletes and cross-cluster events keep the origin's semantics.
* filerstore: return real errors from KvGet on MongoDB, ArangoDB, Elastic
These stores mapped every lookup failure to ErrKvNotFound, so a database error looked identical to a missing key: hard-link records could appear gone while links still owned their chunks, and path timestamp reads could treat committed state as absent. Return the underlying error and keep ErrKvNotFound for genuinely missing keys.
* filer: bind remote-object deletes to the observed object version
A guarded stat-then-delete still lets a replacement landing between the two requests go down with the deleted object. Add an optional conditional-delete operation to remote storage clients — S3 If-Match on the live ETag, GCS generation match, Azure IfMatch — and use it everywhere a delete is guarded by a stored stamp: the write-back daemon's delete, rename-source, and pending-rename paths, and the replayed-delete sweep. Backends without it keep the guarded stat-then-delete.
* filer: restamp the remote entry after a metadata-only remote update
Changing Content-Encoding rewrites the object in place on S3 and advances its LastModified, while the recorded stamp keeps the old mtime. A version-checked delete would then mistake the object for a replacement and keep it forever. Re-stat after UpdateFileMetadata so the saved stamp describes the live object.
* filer: mirror the remote-delete ownership checks in the replayed sweep
The sweep deleted swept children's remote objects on the listing's snapshot
alone. Two gaps: a remote-only entry that took the path since — one a commit
record may not cover, such as a lazy fetch materialized from the remote —
reads the object as its only copy, and an entry removed while its rename
copy was pending leaves the object at the source key orphaned. The per-child
delete now runs under the child's entry lock with the commit-time check
repeated, keeps a remote-only entry that holds the path, and releases the
pending rename's source under the same ownership and stamp conditions the
write-back daemon's releasePendingSource applies.
* remote.sync: restamp only the object the metadata update touched
The live stat after UpdateFileMetadata was saved unconditionally, so a
replacement written between the update and the stat stamped the entry with
an object it never described — and a later delete would then pass the
freshness guard on the replacement's data. The restamp now requires the live
object to still match the stored stamp on size and ETag; a different object
keeps the old stamp, which the delete guard reads as replaced and keeps.
* filer: skip unbudgeted reader-cache test on 32-bit
TestReaderCacheUnboundedWithoutBudget proves the reader cache has no
implicit memory budget by keeping 96 concurrent 4 MiB downloads in flight
(>256 MiB). On linux/386 the weed/filer test binary already runs near the
4 GiB address-space limit, so the burst reliably crashes it with
"out of memory" mid-suite.
* filer: bail out of a swept child lock instead of waiting on it
Replay already holds the event's path locks when the sweep reaches a
descendant. A local rename can hold that descendant while waiting on one
of the replayed paths, so blocking on the child lock deadlocks both. The
sweep now tries the lock and reports ErrLockContended — transient, so the
replication retry replays the event once the local write settles.
* remote: report the stamp a metadata update produced
A metadata write can rewrite the object it touches — S3 changes
Content-Encoding through an in-place copy, which can mint a new ETag —
so restamping from a fresh stat can reject the very object the update
produced, or describe a replacement that landed in between.
UpdateFileMetadata now returns the RemoteEntry of the object it wrote
(S3 from the copy result, GCS from the update attributes, Azure from a
post-update stat); nil means the object was untouched and callers keep
the stored stamp.
* remote: restamp Azure from the metadata write's own etag
* iam: scope CreateServiceAccount's ParentUser by policy resource
AuthorizeIamAction evaluated the iam:CreateServiceAccount grant against a
hardcoded arn:aws:iam:::* resource, so the policy's Resource never had a
say; #11218 then bound the target to the caller outright. Evaluate
self-target actions against the target identity's ARN instead: a wildcard
resource keeps the deliberate any-target grant, a scoped resource confines
issuance to the named parent identity, anything else is denied.
buildResourceARN gains the ARN pass-through buildS3ResourceArn already has,
so iam: ARNs reach the policy engine intact on the attached-policies route.
Closes#11735
* ci: retrigger after unrelated weed/storage metrics flake
* iam: fold IAM-ARN matching into awsIAMMatch, drop debug test
- remove zz_debug_test.go (local debugging leftover)
- fold awsIAMMatchResource into awsIAMMatch: the arn:aws:iam:: value
prefix selects case-sensitive matching, so no separate name is needed
- trim comments
---------
Co-authored-by: batonogov <batonogov@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: resume into the directory a listing marker names
* s3: keep delimited marker resumes exclusive
* s3: descend a marker-named directory without repeating its key
* s3: emit marker-named dir objects but keep internal folders hidden
* webdav: create a folder under its own name
* webdav: reuse NormalizePath for the MKCOL entry split
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The worker fix reads the source's TTL destroy time through
ReadVolumeFileStatus, which returns the in-memory volume_info. Go's
SaveVolumeInfo stamps volumeInfo.ExpireAtSec before writing; Rust's
save_vif wrote it only to the serialized file, so a Rust source kept
reporting the stale creation-time value and the EC .vif would carry a
wrong expiry. Stamp the live volume_info first, mirroring Go and
save_volume_info.
* sts: recognize issuer-scoped parent user ids
ComputeParentUser output has a fixed shape (base64url SHA-256); let
callers distinguish it from bare subject strings.
* s3tables: keep legacy owner re-anchoring off local and scoped account ids
Re-anchoring a stored bare subject under the single trusted issuer must
only apply to strings that could actually be a legacy OIDC subject. A
stored owner naming the reserved or a configured local account, or an
already issuer-scoped id, can otherwise be claimed by any caller whose
token asserts that string as sub.
* s3tables: re-anchor legacy policy principals in map and array forms
A stored bare subject can appear as {"AWS": sub} or inside a principal
list, not only as a top-level string; re-anchoring only the string form
left legacy Deny statements unable to match the caller.
* s3tables: keep built-in legacy exclusions independent of the account checker
A missing local-account predicate must not disable re-anchoring entirely:
the reserved accounts, ARN-shaped values, and scoped-id shapes are never
OIDC subjects regardless, while legacy Deny statements for plain subjects
still need to resolve so a denied caller cannot fall through to an
identity grant.
* s3tables: exercise legacy Deny against an owner grant
The previous case denied the caller regardless of whether the statement
principal matched; make the caller the owner so a missed match shows.
Cover the array Deny form alongside the AWS map.
* s3api: recheck condition under lock when routed filer ignores condition_alt_key
A filer that predates condition_alt_key silently drops the field and
evaluates a suspended-versioning condition against the empty null slot
alone, so a matching If-Match still gets a false 412. Recheck under the
object lock instead of trusting that response: the lock path resolves the
same entry the read path serves. Genuine precondition failures still
return 412 from the recheck.
* s3api: cover suspended If-Match against a filer ignoring condition_alt_key
* s3api: resend against the resolved entry instead of the lock-path recheck
Rechecking under the gateway object lock does not serialize against
routed writers, which commit under the filer's path lock; a concurrent
routed PUT could land between the recheck and the create and be
overwritten with a stale precondition. Resend the same transaction with
condition_key set to the resolved version file so the condition and the
write stay under the same filer lock. When the resolved entry is the
null slot, a delete marker, or unresolvable, the resend targets the
object path and fails closed.
* s3api: resolve the target version in tag-condition policy lookups
* s3api: apply tag-conditioned policy to object deletes
* s3api: apply tag-conditioned policy to object ACL reads and writes
* s3api: apply tag-conditioned policy to retention and legal hold
* s3api: cover tag-conditioned delete, ACL and object-lock rechecks
* s3api: let tag-conditioned grants redecide denied batch deletes
* s3api: resolve the tagged version in delete policy tests
readChunkRange sent the volume lookup's http:// URL through a bare
http.Client, so with [https.client]/[https.volume] configured every read of
an SSE-KMS object larger than one chunk (and every ranged read through this
path) failed with "client sent an HTTP request to an HTTPS server" and the
GET returned 500. Normalise the URL's scheme and use util_http's global
client, which carries the [https.client] TLS configuration, as the other
volume reads already do. Drop the now-unused volumeServerHTTPClient.
Fixes#11799
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: map PRECONDITION_FAILED to a sentinel error
A conditional CreateEntry that the owner's path lock refuses comes back as
FilerError_PRECONDITION_FAILED; mapping it to ErrPreconditionFailed lets the
S3 gateway tell a rejected precondition apart from a transport failure and
report 412 instead of retrying the write.
* s3api: make the ambiguous-PUT recovery create conditional under the lock
createAfterAmbiguousRoute verified state and then issued an unconditional
create, so a write landing between the lookup and the create was still
clobbered by the stale re-commit (issue #11631). The recovery create now
carries a WriteCondition — IF_NOT_EXISTS for a proven-absent entry,
IF_ENTRY_EQUAL for a directory it may need to write onto — which the owner
filer evaluates under its per-path lock, making the check and the write
atomic. A refused precondition loops back to re-read the state instead of
overwriting whatever won the race.
* s3api: do not fail over a routed transaction past a possibly-sent request
withFilerClientFailover retried the callback on the next filer on any
transport error, so an ObjectTransaction whose response was lost could be
re-applied over a write that committed in between (issue #11631). A gRPC
error cannot distinguish "never sent" from "committed", but channel
readiness can: a non-replayable callback now runs only on a connection that
just reached READY, and failover continues only over channels that never
came up — the one failure mode that provably sent nothing.
* s3api: run ambiguous-route recovery for empty PUTs too
The recovery was gated on a nonempty chunk list, so a zero-byte PUT whose
routed transaction lost its response still fell back to an unconditional
create that could overwrite a newer write and delete its chunks. Chunk-count
mismatch on an empty upload list is enough to distinguish a superseding
entry from this PUT's — there is nothing to save by skipping the check.
* s3api: keep conditional recovery creates off the replay path
The conditional create in the ambiguous-PUT fallback rode the replayable
WithFilerClient failover, so a committed create whose response was lost
could be re-sent to another filer — where a delete in between would let
IF_NOT_EXISTS pass again and resurrect an entry pointing at reclaimed
chunks. Conditional creates now run through the non-replayable failover;
an ambiguous result falls through to confirmCreateLanded, which resolves
the committed entry rather than resending.
* s3api: do not mark a filer unhealthy for an answered request
A reply that reached the callback — including an application-level failure
like a refused write precondition — proves the filer is alive. Checking
received before the health bookkeeping keeps definitive application errors
from tripping the failure circuit or flagging a live routed owner as
unreachable.
* s3api: do not claim an empty stored entry as an ambiguous PUT's own
Two chunkless entries compare equal under sameFileChunks, so an older
empty object satisfies "the route landed" even though it predates this
write — recovery would return success while leaving the request's
Content-Type and metadata unapplied. A landed PUT stores the entry
exactly as built, so require mime/size/md5/content and the extended
attributes to match before claiming it; anything else stays uncertain
and the client retries.
* s3api: prove ambiguous empty PUTs by write id and pin recovery to the owner
- Conditional recovery creates are sent to the routed owner alone instead of failing over: the condition only serializes with routed writes under the owner's per-path lock, and another filer's forwarding is only as correct as its ring view. An unreachable owner now yields an uncertain result rather than a create under a different serialization boundary.
- A proven-absent path no longer satisfies an empty, chunkless PUT: the routed commit may have landed and been deleted, and there are no chunks to verify, so IF_NOT_EXISTS could resurrect a removed object. Recovery returns uncertain for the client to retry.
- Chunkless identity moves from metadata equality to a per-write id (x-seaweedfs-write-id) stamped into Extended: an identical older object can match mime/md5/content but not the id. Entries without this write's id are not claimed, and confirmCreateLanded can now positively confirm a committed empty object instead of always reporting failure.
* filer: shrink zip spool threshold in tests to fix 32-bit OOM
TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate
~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the
weed/filer test binary already runs near the 4 GiB address-space limit, so
these allocations reliably crash it with "out of memory" mid-suite.
Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which
exercises the same memory-vs-temp-file spool paths with KiB-sized buffers.
* s3 lifecycle: update stale runShard and WalkerFunc call sites in tests
Recent master changes added a walks param to runShard (#11797) and made WalkerFunc take []int, but replay_view_test.go and run_bounded_test.go still use the old signatures, breaking go vet and test builds on merge.
* fs.meta.load: wait for in-flight entry creation before returning
On io.EOF and error paths the command returned while CreateEntry
goroutines were still running, so trailing entries could be silently
dropped once the shell exited. Wait for them on every exit path and
guard the shared error with a mutex.
* fs.meta.load: surface worker errors on read/unmarshal failure paths
The ReadFull and proto.Unmarshal error paths waited for in-flight
CreateEntry workers but returned only the local error, hiding a failed
create from the caller. Check loadErr after wg.Wait() like the other
exit paths so the worker error is reported.
Extract the load loop into loadEntries so the concurrency and error
propagation can be exercised by unit tests, and add regression tests
covering delayed creates at EOF and on corrupt input plus worker error
reporting.
* s3: count stored versions as bucket contents
A <key>.versions directory holding version entries is user data; skipping
it let bucketHasUserObjects report an empty bucket while the recursive
delete dropped every stored version.
* test: rename stale reserved-folders case in bucket delete test
.versions directories are now descended into, so the case only holds
because the folder is empty.
Chris Lu
github-actions[bot]
