* filer.remote.sync: stamp entries with IF_CHUNKS_EQUAL so a stale write-back cannot delete live chunks
updateLocalEntry records the RemoteEntry stamp after an upload by writing the
event's entry back with UpdateEntry. The filer deletes every stored chunk
absent from an updated entry, so when the file was rewritten while its upload
was in flight (or the event is a replay), the stale snapshot deletes the
rewrite's chunks: the entry then points at the new fid with no needle behind
it, and the rewrite's own upload fails and is skipped as superseded.
The stamp write now carries WriteCondition IF_CHUNKS_EQUAL over the event's
chunk fids, evaluated by the filer under the path lock. A refused stamp means
the filer moved past this event; the superseding event follows in the log and
stamps the current entry, so the refusal is logged and skipped like a
superseded upload.
Reproduction: weed server -filer plus a weed server -s3 remote, remote.mount,
filer.remote.sync; hold the remote (docker pause) so one upload stays in
flight, rewrite the file through the filer, unpause. Before: the entry's chunk
is 404 on every volume server. After: the stale stamp is refused, the rewrite's
chunk stays live and reads back after a vacuum.
* filer.remote.sync: stamp entries with IF_ENTRY_EQUAL so stale inline content or metadata cannot be restored
The IF_CHUNKS_EQUAL guard compared only the chunk fid multiset, so a
rewrite that touched inline content or metadata alone still compared
equal and the stale snapshot overwrote the live entry. The new clause
compares the whole stored entry against the event's entry under the
same path lock.
* filer: route conditional UpdateEntry to the entry's owner filer
Two filers locking the same path locally could still pass a stale
condition on the non-owner while the owner's entry had moved on. When a
condition or expected_extended precondition is set, forward the request
to the entry's owner the same way conditional CreateEntry does, with
is_moved bounding the hop.
* filer: compare IF_ENTRY_EQUAL against the normalized expected entry
FindEntry grows FileSize to the chunk extent, so a raw event entry with
FileSize still zero failed the condition on an unchanged file and the
stamp was skipped, letting a replay upload the object again.
* filer.remote.sync: classify refused stamps by gRPC status only
A FailedPrecondition substring in an unrelated error would have been
swallowed as a skipped stamp; status.FromError already unwraps.
* remote sync: keep the event entry intact for IF_ENTRY_EQUAL
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* filer: accept a WriteCondition on UpdateEntry, under the per-path lock
UpdateEntry was a bare read-modify-write: the precondition check, the
chunk garbage diff, and the store write could interleave with a
concurrent update to the same path. Take the per-path lock CreateEntry
already holds, and evaluate an optional CreateEntry-style WriteCondition
under it, failing with FailedPrecondition like expected_extended.
* filer: IF_CHUNKS_EQUAL write condition compares the stored chunk fid set
A chunk-preserving read-modify-write (tagging, setattr, copy-in-place)
races UpdateEntry's garbage diff: if a concurrent update empties the
chunk list first, the stale writer's commit resurrects fids that are
already queued for deletion, stranding the entry on a dead needle once
vacuum reclaims it. The reverse also holds: a writer that read an empty
chunk list can wipe chunks a concurrent update just added.
IF_CHUNKS_EQUAL guards both: the stored chunk fid multiset must still
equal what the caller read, order-independent, with an empty fids list
expecting no chunks. Absent entry counts as no chunks for CreateEntry
overwrites and transactions.
* filer: delete and append serialize on the entry path lock
DeleteEntry queues the entry's chunks for deletion and AppendToEntry
rewrites the chunk list, but neither held the per-path lock, so either
could interleave with a conditional update between its precondition
check and its write — a passed IF_CHUNKS_EQUAL would then resurrect
fids already on the deletion queue, or clobber a freshly appended
chunk. AppendToEntry keeps the cluster lock for cross-filer append
serialization; the path lock covers the local read-modify-write.
* filer: reuse lockPath in UpdateEntry lookup
A version-specific DELETE (real version or the null version, including
object-lock WORM-checked ones and governance-bypass) now runs as one routed
transaction on the object's owner instead of holding the distributed lock.
For a real version: recompute the .versions pointer excluding the version
(repoint-before-delete, so a crash leaves a recoverable orphan rather than a
dangling pointer), then delete the version file, under the object's per-path lock.
The null version is the regular object entry, deleted directly (no pointer).
Object-lock buckets gate the delete on the version's WORM guards evaluated on the
owner: legal hold (always) + retention (while not elapsed). Governance bypass
scopes the retention guard to COMPLIANCE mode, so the filer allows a
governance-mode delete while still denying compliance and legal hold — the
gateway never reads the version.
Three primitives make this expressible:
- ObjectTransaction.condition_key: evaluate the condition against a named entry
(the version) while the lock stays on lock_key (the object).
- Recompute.exclude_name: omit a child from the scan, to repoint before delete.
- WriteCondition.Clause gate_key/gate_value: scope IF_EXTENDED_TIME_ELAPSED to a
mode, expressing governance bypass without a gateway-side read.
Routing object-lock buckets off the distributed lock needs the retention and
legal-hold check to run atomically with the write, under the per-path lock. Move
just the comparison into the filer, not the S3 semantics: two generic clause
kinds on an extended attribute.
IF_EXTENDED_NOT_EQUAL blocks while extended[ext_key] equals ext_value (a legal
hold). IF_EXTENDED_TIME_ELAPSED blocks while extended[ext_key], read as a unix-
second deadline, is in the future against the filer's clock (retention); a
malformed deadline fails safe. The caller composes these from the object-lock
state and, for a governance bypass, simply omits the retention clause once the
bypass is authorized -- the filer makes no authorization decision and keeps no
S3 knowledge.
Add an optional WriteCondition to CreateEntryRequest. When set, the filer
evaluates it against the current entry while holding the per-path lock, so the
check and the write are atomic on this filer, and returns PRECONDITION_FAILED
when it does not hold. The caller must route the key's writes to the owner filer
for the check to be authoritative.
A condition is a list of clauses that all must hold (logical AND). One clause is
the common case; several express what a single comparison cannot: an ETag set
(If-Match / If-None-Match with multiple values), weak-ETag comparison, and
compound conditions. ETag comparison mirrors the S3 gateway's precedence (stored
Seaweed ETag attribute, then the Md5/chunk fallback) and follows RFC 7232
strong/weak rules, so results match without coupling the filer to S3 handling.
Condition parsing and evaluation live in filer_grpc_server_condition.go.
