* cluster: never broadcast an empty lock ring
An empty member list is never a usable ring state, but a delayed
RemoveServer on a former leader can fire after the new leader already
broadcast the recovered ring. That late broadcast carries a newer
wall-clock version, so clients accept the empty ring and permanently
reject the good one.
Skip the broadcast entirely when the member list is empty, keeping the
last non-empty snapshot for reconnecting clients.
* cluster: periodically rebroadcast the lock ring
Ring updates are purely event-driven, so one lost or poisoned update is
permanent until the next membership change — with a single filer that may
never come. Re-arm a per-group timer after every broadcast so the current
leader keeps re-sending the ring; clients reject nothing newer than their
last accepted version, so a re-sent snapshot always heals a stale view.
* filer,s3api: reset the lock ring on master change
Ring versions are per-master monotonic — each master stamps wall-clock
nanoseconds — so a late high-version update accepted from a former leader
makes the new leader's snapshot look stale forever. Detect a leader
change across the reconnect gap (currentMaster is cleared between
attempts, so remember the last served master) and reset the ring to
bootstrap state so the new leader's view always applies.
* cluster: fail lock acquisition when no lock server exists
retryUntilLocked loops forever, so a filer reporting an empty lock ring
wedges every append write indefinitely. Bound only the "no lock server
found" case — ordinary contention is still waited out since the holder
releases eventually. The constructors now return nil on failure: the
filer append path and S3 object writes fail fast, while mounts degrade
to their existing lockless mode.
* cluster: reset only the ring version on master change
Ring versions are per-master monotonic, so a version gate reset is all a
leader change needs. Clearing the whole ring made every filer its own
write owner until the next update and dropped the prior-owner window for
keys the new leader remaps; the last ring now keeps routing until the
new leader's snapshot transitions off it.
* cluster: skip redundant ring installs and defer rebroadcasts
An unchanged member list now only bumps the accepted version instead of
installing a snapshot: periodic rebroadcasts no longer fire the
topology-change callback or restart the prior-owner window. And a
rebroadcast that lands inside a membership stabilization window yields
to the pending timer rather than publishing an intermediate ring.
* cluster,mount: bound lock unavailability, fail ops that cannot lock
Only 'lock already owned' contention retries without bound now; every
other failure — no lock server, or a dead ring member refusing
connections — shares the same unavailability budget, so a ring naming
departed filers can no longer hang a lock forever. Mount open-write,
create, and rename fail with EAGAIN when the required lock cannot be
acquired instead of proceeding without cross-mount serialization.
* cluster: check pending stabilization inside the broadcast critical section
rebroadcast released the mutex between the pending-timer check and
nextBroadcastUpdate, so a membership change arriving in the gap could arm
a stabilization timer while the rebroadcast emitted an intermediate ring.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: acquire path locks before mutating create/rename state
Create took the DLM lock only after the filer create, so a lock failure
returned EAGAIN with an eagerly persisted file left behind. Rename marked
source handles renamed before acquiring locks, so a failed acquisition
left them suppressing old-path flushes for a rename that never happened.
Both now take the locks first; the create's lock is released again if the
entry race loses to another creator and AcquireHandle takes over.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: keep the old-path lock when rename lock migration fails
The migration stopped the handle's lock before acquiring the replacement,
so a nil result left the handle writing with no lock at all. Acquiring the
new-path lock first means failure keeps the existing lock instead of
reporting success with serialization dropped.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: skip new-path rename lock when a handle already holds it
A target file open for write on this mount already carries a lock on
newPath; the lock manager does not grant a second lock to the same
owner, so the rename would wait on itself until the handle closed.
Also avoid locking twice when old and new paths coincide.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: hand the rename's target lock to the migrating handle
The rename holds a lock on newPath for its duration, so the response
migration's fresh acquisition waited on that same lock until the handle
released — under fhLockTable, blocking the handle's own close. Adopt the
rename's lock directly; nested move responses still acquire their own.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: move the replaced target's lock to the renamed handle
When the target path was already locked by an open handle on this
mount, the migrated source handle kept only its stale old-path lock —
the target's close would then release the last lock on the new path
while the renamed handle was still open. Adopt the replaced handle's
lock instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: stop the handle lock inside the fh lock on release
ReleaseHandle stopped fh.dlmLock before taking the fhLockTable slot, so
a rename migration holding that slot could still observe and adopt a
lock that was already stopping. Stopping under the fh lock makes the
transfer serialize against the release.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: claim the replaced target's lock for the renamed handle
When the target path is already locked by an open handle on this mount,
adopting it at migration time keeps the renamed path protected after
that handle closes, without waiting on a lock this mount already holds.
If the handle was released mid-migration the claimed lock is stopped,
and a fresh acquire covers the case where it was already gone.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: claim the target handle's lock before the rename runs
Skipping the new-path lock when a handle already holds it let that
handle's close release the lock mid-rename, leaving the path unguarded
until the response migrated it. Take over the lock at check time and
hold it for the rename's duration: the response adopts it for the
migrating handle, or it returns to the target handle / is released on
failure. The target handle lookup also falls back to the entry's stored
inode for a forgotten path mapping.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: read handle locks only under the fh lock during rename
The loose dlmLock reads raced ReleaseHandle, which now mutates the lock
inside the handle lock; check and claim it under the same hold.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
ObjectTransaction forwards to the ring owner so one filer's per-path lock
arbitrates every writer of a key. But a ring change hands the key over before
the new owner has rebuilt the locks the prior owner still holds, so for the
cooling-off window both can grant it. LockRing.PriorOwner exists for exactly
this and nothing consulted it.
Route to the prior owner while that window is open. LockRing.WriteOwner
resolves prior-else-current under one read lock, so the pair cannot come from
different rings and name the same filer twice.
An unreachable owner fails the request rather than falling back to the current
one. gRPC reports a response lost in transit as Unavailable, indistinguishable
from a request the owner never saw, so re-sending elsewhere could re-apply what
the owner already committed; and an owner unreachable from here may be
partitioned rather than down, still serving the key to everyone else — which is
the split brain the routing exists to prevent. The window is bounded: once it
closes the ring hands the key to its new owner.
The owner resolution and the forward move into writeOwner/forwardToWriteOwner
so the next routed RPC reuses them rather than copying the block.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
While the ring changed within the last snapshot interval, a fresh owner
asks the key's previous owner (LockRing.PriorOwner) whether it still
holds a conflicting lock before granting TRY_LOCK or answering GET_LK, so
it does not double-grant before re-assertion rebuilds its local state.
The probe is marked cooling_probe so the previous owner answers from
local state without recursing. PriorOwner uses the snapshot's prebuilt
ring rather than rebuilding a hash ring per call.
* s3: dial the object lock's primary filer directly
The S3 object write lock builds a fresh short-lived lock per write, each
starting at the seed filer. When the seed isn't the key's hash-ring primary
the filer forwards the request to the primary, and in multi-cluster setups
that forward crosses clusters on every write.
Give the lock client a view of the filer lock ring, fed by the master's
LockRingUpdate broadcasts the gateway already receives, so it dials the
primary directly. The view tracks filer membership by version; a stale view
stays correct because the filer still forwards as a fallback.
Also send the initial ring snapshot to S3 clients, not just filers.
* s3: subscribe to lock-ring updates before starting the master loop
The master delivers the initial LockRingUpdate once, on connect. Registering the
callback after KeepConnectedToMaster started left a window where that first
update could arrive before the handler was set and be dropped, delaying the ring
view until the next membership change. Build the lock client and register the
callback in the masters block before launching the loop; the filers block reuses
that client (or creates a plain one when no masters are configured).
* lock_manager: build the hash ring in a deterministic server order
rebuildRing ranged over the server set (a map), whose iteration order is
randomized per process. On a vnode hash collision the last writer into
vnodeToServer wins, so two nodes holding the same server set could resolve the
collision to different servers and disagree on the primary for keys near that
slot. Now that the S3 gateway also computes PrimaryForKey, such a disagreement
would route the same key to different filers and defeat per-path serialization.
Iterate the servers in sorted order so the ring is identical on every node with
the same set, regardless of discovery order.
* lock_manager: skip redundant ring rebuilds, trim comments
SetRing now ignores a non-zero version at or below the current one once a ring
exists, so repeated LockRingUpdate broadcasts on reconnect no longer rebuild the
ring.
* s3: hold the lock-ring client on the server for route-by-key
Store the object-write lock client on S3ApiServer so handlers can resolve a
key's owner filer via PrimaryForKey.
* dlm: replace modulo hashing with consistent hash ring
Introduce HashRing with virtual nodes (CRC32-based consistent hashing)
to replace the modulo-based hashKeyToServer. When a filer node is
removed, only keys that hashed to that node are remapped to the next
server on the ring, leaving all other mappings stable. This is the
foundation for backup replication — the successor on the ring is
always the natural takeover node.
* dlm: add Generation and IsBackup fields to Lock
Lock now carries IsBackup (whether this node holds the lock as a backup
replica) and Generation (a monotonic fencing token that increments on
each fresh acquisition, stays the same on renewal). Add helper methods:
AllLocks, PromoteLock, DemoteLock, InsertBackupLock, RemoveLock, GetLock.
* dlm: add ReplicateLock RPC and generation/is_backup proto fields
Add generation field to LockResponse for fencing tokens.
Add generation and is_backup fields to Lock message.
Add ReplicateLock RPC for primary-to-backup lock replication.
Add ReplicateLockRequest/ReplicateLockResponse messages.
* dlm: add async backup replication to DistributedLockManager
Route lock/unlock via consistent hash ring's GetPrimaryAndBackup().
After a successful lock or unlock on the primary, asynchronously
replicate the operation to the backup server via ReplicateFunc
callback. Single-server deployments skip replication.
* dlm: add ReplicateLock handler and backup-aware topology changes
Add ReplicateLock gRPC handler for primary-to-backup replication.
Revise OnDlmChangeSnapshot to handle three cases on topology change:
- Promote backup locks when this node becomes primary
- Demote primary locks when this node becomes backup
- Transfer locks when this node is neither primary nor backup
Wire up SetupDlmReplication during filer server initialization.
* dlm: expose generation fencing token in lock client
LiveLock now captures the generation from LockResponse and exposes it
via Generation() method. Consumers can use this as a fencing token to
detect stale lock holders.
* dlm: update empty folder cleaner to use consistent hash ring
Replace local modulo-based hashKeyToServer with LockRing.GetPrimary()
which uses the shared consistent hash ring for folder ownership.
* dlm: add unit tests for consistent hash ring
Test basic operations, consistency on server removal (only keys from
removed server move), backup-is-successor property (backup becomes
new primary when primary is removed), and key distribution balance.
* dlm: add integration tests for lock replication failure scenarios
Test cases:
- Primary crash with backup promotion (backup has valid token)
- Backup crash with primary continuing
- Both primary and backup crash (lock lost, re-acquirable)
- Rolling restart across all nodes
- Generation fencing token increments on new acquisition
- Replication failure (primary still works independently)
- Unlock replicates deletion to backup
- Lock survives server addition (topology change)
- Consistent hashing minimal disruption (only removed server's keys move)
* dlm: address PR review findings
1. Causal replication ordering: Add per-lock sequence number (Seq) that
increments on every mutation. Backup rejects incoming mutations with
seq <= current seq, preventing stale async replications from
overwriting newer state. Unlock replication also carries seq and is
rejected if stale.
2. Demote-after-handoff: OnDlmChangeSnapshot now transfers the lock to
the new primary first and only demotes to backup after a successful
TransferLocks RPC. If the transfer fails, the lock stays as primary
on this node.
3. SetSnapshot candidateServers leak: Replace the candidateServers map
entirely instead of appending, so removed servers don't linger.
4. TransferLocks preserves Generation and Seq: InsertLock now accepts
generation and seq parameters. After accepting a transferred lock,
the receiving node re-replicates to its backup.
5. Rolling restart test: Add re-replication step after promotion and
assert survivedCount > 0. Add TestDLM_StaleReplicationRejected.
6. Mixed-version upgrade note: Add comment on HashRing documenting that
all filer nodes must be upgraded together.
* dlm: serve renewals locally during transfer window on node join
When a new node joins and steals hash ranges from surviving nodes,
there's a window between ring update and lock transfer where the
client gets redirected to a node that doesn't have the lock yet.
Fix: if the ring says primary != self but we still hold the lock
locally (non-backup, matching token), serve the renewal/unlock here
rather than redirecting. The lock will be transferred by
OnDlmChangeSnapshot, and subsequent requests will go to the new
primary once the transfer completes.
Add tests:
- TestDLM_NodeDropAndJoin_OwnershipDisruption: measures disruption
when a node drops and a new one joins (14/100 surviving-node locks
disrupted, all handled by transfer logic)
- TestDLM_RenewalDuringTransferWindow: verifies renewal succeeds on
old primary during the transfer window
* dlm: master-managed lock ring with stabilization batching
The master now owns the lock ring membership. Instead of filers
independently reacting to individual ClusterNodeUpdate add/remove
events, the master:
1. Tracks filer membership in LockRingManager
2. Batches rapid changes with a 1-second stabilization timer
(e.g., a node drop + join within 1 second → single ring update)
3. Broadcasts the complete ring snapshot atomically via the new
LockRingUpdate message in KeepConnectedResponse
Filers receive the ring as a complete snapshot and apply it via
SetSnapshot, ensuring all filers converge to the same ring state
without intermediate churn.
This eliminates the double-churn problem where a rapid drop+join
would fire two separate ring mutations, each triggering lock
transfers and disrupting ownership on surviving nodes.
* dlm: track ring version, reject stale updates, remove dead code
SetSnapshot now takes a version parameter from the master. Stale
updates (version < current) are rejected, preventing reordered
messages from overwriting a newer ring state. Version 0 is always
accepted for bootstrap.
Remove AddServer/RemoveServer from LockRing — the ring is now
exclusively managed by the master via SetSnapshot. Remove the
candidateServers map that was only used by those methods.
* dlm: fix SelectLocks data race, advance generation on backup insert
- SelectLocks: change RLock to Lock since the function deletes map
entries, which is a write operation and causes a data race under RLock.
- InsertBackupLock: advance nextGeneration to at least the incoming
generation so that after failover promotion, new lock acquisitions
get a generation strictly greater than any replicated lock.
- Bump replication failure log from V(1) to Warningf for production
visibility.
* dlm: fix SetSnapshot race, test reliability, timer edge cases
- SetSnapshot: hold LockRing lock through both version update and
Ring.SetServers() so they're atomic. Prevents a concurrent caller
from seeing the new version but applying stale servers.
- Transfer window test: search for a key that actually moves primary
when filer4 joins, instead of relying on a fixed key that may not.
- renewLock redirect: pass the existing token to the new primary
instead of empty string, so redirected renewals work correctly.
- scheduleBroadcast: check timer.Stop() return value. If the timer
already fired, the callback picks up latest state.
- FlushPending: only broadcast if timer.Stop() returns true (timer
was still pending). If false, the callback is already running.
- Fix test comment: "idempotent" → "accepted, state-changing".
* dlm: use wall-clock nanoseconds for lock ring version
The lock ring version was an in-memory counter that reset to 0 on
master restart. A filer that had seen version 5 would reject version 1
from the restarted master.
Fix: use time.Now().UnixNano() as the version. This survives master
restarts without persistence — the restarted master produces a
version greater than any pre-restart value.
* dlm: treat expired lock owners as missing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: reject stale lock transfers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: order replication by generation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: bootstrap lock ring on reconnect
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Chris Lu
M@andGitHub