Conditional PUTs require a lock primitive that excludes competing gateway
processes sharing a backend filesystem. A successful flock call does not prove
that property: some clustered filesystem configurations accept flock but scope
it to one node, silently leaving cross-gateway check-and-publish races open.
Add an object-lock mode that lets operators select flock or fcntl for
filesystems where that primitive is cluster-coherent, local for the existing
per-process behavior, or none to reject conditional writes with NotImplemented.
Keep the legacy disable flag as an alias for local.
Shared lock modes now verify the selected primitive on the root lock filesystem
during startup and fail closed if it cannot be used. Runtime lock failures no
longer silently downgrade to process-local exclusion. The startup check cannot
establish cross-node coherence, so that remains an explicit operator
requirement.
ScoutFS defaults to none since posix locks are not cluster consistent, but
allow setting local for single node deployments.
Fixes#2351
Fail closed on unexpected advisory-lock errors instead of silently reducing
cross-process exclusion to a local mutex. Make local publish-slot waits honor
request cancellation. Move version snapshots under the publish lock so
concurrent versioned PUTs preserve publication order. Add regression coverage
for canceled lock waiters.
Also add an option to disable flock files and only rely on in process locking.
S3 conditional writes (If-Match / If-None-Match: *) were evaluated with a
check-then-act pattern: PutObjectWithPostFunc and
CompleteMultipartUploadWithCopy read the current etag, evaluated the
precondition, and only later published the replacement via link/rename.
Under concurrent conditional PUTs to the same key, many writers could read
the same old state, all pass the check, and all succeed, breaking
compare-and-swap coordination built on conditional writes (observed by
Buzz's git object-store A3 conformance probe: 32-way races returned up to
32 winners instead of exactly 1x2xx + 31x412).
Introduce a per-object publish lock (lockObjectPublish) that every object
publication path holds across condition re-evaluation, metadata stores,
and the final link:
- Exclusion is an advisory file lock (flock on unix, LockFileEx on
windows) on bucket/.sgwtmp/objlock/<shard>, where shard is the first
byte of sha256(key). The gateway is stateless and multiple gateway
processes may share one filesystem, so a process-local mutex alone is
not sufficient; file locks provide cross-process and (via NFSv4 lock
semantics) cross-client exclusion. A process-local striped mutex is
taken alongside so in-process contention never thrashes the filesystem
lock. Lock files are empty, bounded (max 256 per bucket), and never
unlinked to avoid the unlink/recreate flock race that cannot be
detected reliably on NFS.
- Request bodies are staged to the temp file before the lock is taken;
the lock covers only the short commit phase. The kernel releases the
lock on close or process death, so failures and crashes cannot leave a
stale lock. If the filesystem does not support advisory locking (e.g.
NFS mounted with -o nolock), the gateway falls back to process-local
exclusion and warns once.
- The pre-staging precondition check is kept as an advisory fast-fail;
the authoritative check runs under the lock. Unconditional PUTs,
directory-object PUTs, CopyObject (via PutObject), scoutfs (via
PutObjectWithPostFunc/CompleteMultipartUploadWithCopy), and multipart
completion all participate.
- For conditional writes on versioned buckets the version snapshot is
deferred until the precondition is confirmed under the lock, so losing
writers no longer create spurious version snapshots.
- The postprocess hook now runs before any path-based metadata is
written, so a failing hook no longer leaves stale sidecar attributes
(previously the new etag was visible in sidecar mode even when
publication failed).
Add regression tests covering 32-way If-Match and If-None-Match: * races
for both ordinary PUT and multipart completion, mixed
conditional/unconditional races, sequential semantics, per-key
independence, and cleanup after failed publication, for both xattr and
sidecar metadata modes. Add an external SigV4 test
(tests/conditionalrace) that replicates the Buzz A3 probe against a live
gateway, optionally across two gateway processes sharing one backend
directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ben McClelland
Carl MercierandClaude Fable 5