A newly elected leader fences the previous quorum leader as it's removed
from the quorum, and the same rid is also a mounted client that the new
server will scan and prepare for recovery. If that fence is still pending
when the 30s recovery timeout fires, the rid is fenced a second time and
the two fence requests collide on the rid-named sysfs dir.
Close the window at its source: reclaim the pending fences and wait for
them to fully drain before scanning the mounted client btree for
recovery. Reclaim removes a fenced rid from that btree, so once the fence
list is empty the previous leader is no longer a recovery candidate and
the recovery timeout can't fence it again.
Add scoutfs_fence_drained() as a non-blocking predicate in the fence
layer: it reports an empty pending list and surfaces any errored fence.
The server owns the wait in wait_for_fence_drain(), blocking on
server->waitq so it wakes promptly when the server stops.
queue_reclaim_work() moves ahead of start_recovery() so reclaim runs
during the drain.
Draining is only an optimization, since the existing fence dedup already
makes a double fence non-fatal. The backstop timeout is therefore best
effort: on a large system a single reclaim can legitimately run longer
than the timeout, and aborting a healthy reclaim would recreate the
recovery-failure loop we're avoiding. On timeout the server warns and
starts recovery anyway, letting reclaim finish in the background and dedup
absorb any double-submit. Startup is only aborted if a fence errored (the
node couldn't be fenced, so recovery isn't safe) or the server is already
stopping.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Auke Kok
4a8d2211fe