mirror of
https://github.com/versity/versitygw.git
synced 2026-09-29 03:05:34 +00:00
fix(posix): make conditional PUT evaluation and publication atomic per key
S3 conditional writes (If-Match / If-None-Match: *) were evaluated with a check-then-act pattern: PutObjectWithPostFunc and CompleteMultipartUploadWithCopy read the current etag, evaluated the precondition, and only later published the replacement via link/rename. Under concurrent conditional PUTs to the same key, many writers could read the same old state, all pass the check, and all succeed, breaking compare-and-swap coordination built on conditional writes (observed by Buzz's git object-store A3 conformance probe: 32-way races returned up to 32 winners instead of exactly 1x2xx + 31x412). Introduce a per-object publish lock (lockObjectPublish) that every object publication path holds across condition re-evaluation, metadata stores, and the final link: - Exclusion is an advisory file lock (flock on unix, LockFileEx on windows) on bucket/.sgwtmp/objlock/<shard>, where shard is the first byte of sha256(key). The gateway is stateless and multiple gateway processes may share one filesystem, so a process-local mutex alone is not sufficient; file locks provide cross-process and (via NFSv4 lock semantics) cross-client exclusion. A process-local striped mutex is taken alongside so in-process contention never thrashes the filesystem lock. Lock files are empty, bounded (max 256 per bucket), and never unlinked to avoid the unlink/recreate flock race that cannot be detected reliably on NFS. - Request bodies are staged to the temp file before the lock is taken; the lock covers only the short commit phase. The kernel releases the lock on close or process death, so failures and crashes cannot leave a stale lock. If the filesystem does not support advisory locking (e.g. NFS mounted with -o nolock), the gateway falls back to process-local exclusion and warns once. - The pre-staging precondition check is kept as an advisory fast-fail; the authoritative check runs under the lock. Unconditional PUTs, directory-object PUTs, CopyObject (via PutObject), scoutfs (via PutObjectWithPostFunc/CompleteMultipartUploadWithCopy), and multipart completion all participate. - For conditional writes on versioned buckets the version snapshot is deferred until the precondition is confirmed under the lock, so losing writers no longer create spurious version snapshots. - The postprocess hook now runs before any path-based metadata is written, so a failing hook no longer leaves stale sidecar attributes (previously the new etag was visible in sidecar mode even when publication failed). Add regression tests covering 32-way If-Match and If-None-Match: * races for both ordinary PUT and multipart completion, mixed conditional/unconditional races, sequential semantics, per-key independence, and cleanup after failed publication, for both xattr and sidecar metadata modes. Add an external SigV4 test (tests/conditionalrace) that replicates the Buzz A3 probe against a live gateway, optionally across two gateway processes sharing one backend directory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
8bd73a45ee
commit
f7e13d71ce
@@ -0,0 +1,51 @@
|
||||
// Copyright 2026 Versity Software
|
||||
// This file is licensed under the Apache License, Version 2.0
|
||||
// (the "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
//go:build !windows
|
||||
|
||||
package posix
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
// lockFileExclusive takes an exclusive advisory flock on f, polling with
|
||||
// backoff so that context cancellation is honored while waiting. The lock is
|
||||
// released by closing f or on process exit, so no cleanup beyond Close is
|
||||
// required on any failure path.
|
||||
func lockFileExclusive(ctx context.Context, f *os.File) error {
|
||||
backoff := objLockInitialBackoff
|
||||
for {
|
||||
err := unix.Flock(int(f.Fd()), unix.LOCK_EX|unix.LOCK_NB)
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
if !errors.Is(err, unix.EWOULDBLOCK) && !errors.Is(err, unix.EAGAIN) &&
|
||||
!errors.Is(err, unix.EINTR) {
|
||||
return err
|
||||
}
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(backoff):
|
||||
}
|
||||
backoff = min(backoff*2, objLockMaxBackoff)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user