* s3: fail over routed object writes when the owner filer is unreachable
A routed object write (multipart completion, PUT, delete, versioned
finalize, metadata replace) dialed the ring-selected owner filer
directly with no failover. After a filer restarts onto a new address the
lock ring can still name the old one, so every routed write hangs on the
dead address until the gateway is restarted; CompleteMultipartUpload in
particular exceeds client timeouts.
Route the transaction through withFilerClientFailover, skipping an owner
that recently failed, so a live filer forwards it to the real owner by
route_key. Mirrors the read path's getObjectEntryRoutedByKey.
* s3: fail over bucket-config writes when the owner filer is unreachable
patchBucketEntry dialed the bucket's ring owner directly, so a restarted
filer's stale ring address hung every bucket-config write (versioning,
lifecycle, object lock, ownership, ACL, policy, CORS) - the same failure
as routed object writes. Route it through objectTxnOnFiler so it skips an
unreachable owner and a live filer forwards by route_key.
* s3: commit a versioned PutObject and its latest pointer in one transaction
A versioned PutObject wrote the version file and flipped the .versions
latest pointer in two separate routed transactions. Fold the
RECOMPUTE_LATEST into the version file's PUT so both commit atomically
under the object's per-path lock: the recompute, applied after the PUT in
the same transaction, scans the directory and sees the new version. A
crash can no longer leave the version present with a stale pointer.
putToFiler now takes a putFinalize describing the finalize step — routed
mutations folded into the PUT, or an afterCreate run under the object
write lock off the ring. Suspended-versioning keeps its afterCreate-only
form; multipart, copy, and delete-marker finalizes are unchanged.
* s3: trim verbose finalize comments
A non-owner filer forwards the whole transaction to the ring owner of route_key, so the owner's per-path lock stays the single serialization point even when the caller's ring view is stale. is_moved bounds forwarding to one hop. The gateway stamps route_key on every routed builder via the shared objectRouteKey helper. Completes taking S3 object mutations off the distributed lock.
A non-versioned metadata-only self-copy (CopyObject with source == destination
and the REPLACE directive) is a read-modify-write of one entry, which is why it
held the distributed lock. It now routes to the owner as a serialized
PATCH_EXTENDED: the owner merges the new managed metadata (set the replacements,
delete the dropped keys) onto a fresh read of the entry under its per-path lock,
so a concurrent change to non-managed keys (legal hold, retention, version id) is
preserved instead of clobbered, and bumps mtime.
PATCH_EXTENDED gains touch_mtime for the mtime bump. Versioned and suspended
self-copies create a new version (already routed via the copy finalize) and the
no-owner bootstrap keep the lock.
completeMultipartUpload routes its writes to the object's owner filer when an
owner is known, off the distributed lock. Idempotent replay is handled
gateway-side in prepareMultipartCompletionState (it returns the existing result
when the object already carries this UploadId), so the lock is not needed to
dedupe retries; with no owner yet, the lock remains as the bootstrap path.
Versioned completion flips the .versions pointer via routedVersionedFinalize
(RECOMPUTE_LATEST). Non-versioned and suspended completion write the object via
routedMkFile (a routed PUT) so the write serializes with concurrent writes to
the same key on the owner's per-path lock. The version file itself is a unique
path and stays a plain mkFile.
routableWriteOwner no longer excludes object-lock buckets, so a versioned PUT
(which creates a new version, never overwriting a locked one) and a
non-versioned overwrite (WORM-checked gateway-side before dispatch) route to the
owner filer like any other write.
routedObjectOwner still excludes object-lock: an unversioned object-lock delete
enforces WORM under the lock, so it stays there rather than routing past the
check. Version-specific deletes likewise stay on the lock — routing them needs
the WORM check (on the version entry) and the latest-pointer recompute (on the
object) under one transaction, which the current single condition target cannot
express.
s3: route versioned PutObject finalize off the distributed lock
A versioned write's finalize (flip the .versions pointer to the newest version,
demote the prior latest) now runs as a single RECOMPUTE_LATEST ObjectTransaction
on the object's owner filer, under its per-path lock, instead of the unserialized
updateLatestVersionInDirectory. The version file is written first; the owner
re-derives the pointer by scanning the directory.
RECOMPUTE_LATEST gains size_to_key / mtime_to_key to cache the chosen version's
size and mtime on the pointer, and demote_key / demote_value to stamp the
displaced prior latest (NoncurrentSinceNs for lifecycle) when the pointer moves.
Falls back to updateLatestVersionInDirectory when no owner is known yet.
s3: route non-versioned object PUT and DELETE off the distributed lock
A non-versioned, non-object-lock object write now goes straight to the key's
owner filer as a single-mutation ObjectTransaction, which serializes it with the
owner's per-path lock and evaluates the precondition, instead of taking a
cluster-wide lock. PUT and DELETE use the object's full path as the lock key, so
a concurrent create and delete of the same key serialize against each other.
The fast path is taken only when the precondition reduces to clauses the filer
can evaluate (existence and a single strong-ETag match); time-based conditions,
ETag lists, weak ETags, post-create hooks, and an unknown owner fall back to the
lock. A routed mutation error other than a failed precondition also falls back,
so the lock path stays the authority for the cases it alone covers.
PrimaryForKey returns "" until the ring view arrives, keeping writes on the lock
until routing is known.
Chris LuandGitHub