From 313674a439e63675ebfd1991e56f4bd3fb2b28dd Mon Sep 17 00:00:00 2001 From: Auke Kok Date: Mon, 15 Jun 2026 11:37:48 -0700 Subject: [PATCH] Don't shut down the server when fencing a rid twice A node only needs to be fenced once, but scoutfs_fence_start() can be called for the same rid more than once. When a new leader starts it fences the previous leader as it removes it from the quorum (quorum_block_leader), and that same rid can also be a mounted client that then fails to recover within the timeout (client_recovery). The second fence call collides on that name, sysfs returns -EEXIST, and the error is propagated to fence_pending_recov_worker() which treats any error as fatal and shuts the server down. On the next mount a new leader hits the same stale set and the same collision, so the filesystem can never finish recovery. Jun 15 09:22:35 kernel: scoutfs f.000000.r.222222: fencing previous leader f.000000.r.111111 at term 183942 in slot 3 with address x.x.x.x:6000 Jun 15 09:22:36 scoutfs-fenced[9194]: [2026-06-15 09:22:36.588037842] server f.000000.r.222222 fencing rid 1111111111111111 at IP x.x.x.x for quorum_block_leader Jun 15 09:23:09 kernel: scoutfs f.000000.r.222222 error: 30000 ms recovery timeout expired for client rid 1111111111111111, fencing Jun 15 09:23:09 kernel: sysfs: cannot create duplicate filename '/fs/scoutfs/f.000000.r.222222/fence/1111111111111111' Jun 15 09:23:09 kernel: scoutfs f.000000.r.222222 error: fence returned err -17, shutting down server The reclaim path keys off the rid, not the reason, so one pending fence covers both needs. Skip submitting a duplicate fence for an rid that already has one pending, and treat a -EEXIST from the sysfs create as success to also cover a race or a stale rid-named dir. Signed-off-by: Auke Kok --- kmod/src/fence.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/kmod/src/fence.c b/kmod/src/fence.c index c63f27b4..86c7ff65 100644 --- a/kmod/src/fence.c +++ b/kmod/src/fence.c @@ -238,6 +238,25 @@ int scoutfs_fence_start(struct super_block *sb, u64 rid, __be32 ipv4_addr, int r struct pending_fence *fence; int ret; + /* + * A node only needs to be fenced once. A given rid can be submitted + * for fencing more than once: the previous quorum leader is fenced as + * it's removed from the quorum, and that same rid can also be a + * mounted client that fails to recover in time. The reclaim path + * keys off the rid, not the reason, so a single pending fence is + * sufficient. Skip creating a duplicate, which would otherwise + * collide on the rid-named sysfs dir and return -EEXIST to the caller, + * shutting the server down. + */ + spin_lock(&fi->lock); + list_for_each_entry(fence, &fi->list, entry) { + if (fence->rid == rid) { + spin_unlock(&fi->lock); + return 0; + } + } + spin_unlock(&fi->lock); + fence = kzalloc(sizeof(struct pending_fence), GFP_NOFS); if (!fence) { ret = -ENOMEM; @@ -258,6 +277,9 @@ int scoutfs_fence_start(struct super_block *sb, u64 rid, __be32 ipv4_addr, int r &fence->ssa, fence_attrs, "%016llx", rid); if (ret < 0) { + /* highly unlikely race collision */ + if (ret == -EEXIST) + ret = 0; kfree(fence); goto out; }