Avoid runaway election terms in severed nodes.

A server that is severed from the rest of the cluster will notice
that it's peers are no longer present. When it does, it will fall
back to becoming a follower, and attempt to rebuild quorum.

It does so by calling elections. Into the void. None of the elections
will receive answers. Each election cycle lasts sub-seconds, so the
TERM is durably increased many times in the span of seconds.

The remainder of the cluster will reform quorum and after about 30s,
fence the original leader. In many cases, the old leader will have
started these futile elections already before the fence completely
shuts down the original server.

When the server is rebooted, it will read it's quorum blocks
and continue off with the inflated TERM value. This causes it to
immediately become the new leader of the cluster, because it's TERM
is greatly beyond the other nodes, which just recovered quorum with
a single TERM increase.

In a single FS environment, this is all fine with a 3-node cluster. At
no point in time is the cluster degraded, but, there is an unnecessary
fence in this sequence: the second fence disrupts a healthy cluster.
Recovery from this second fence is normal. No further TERM inflation
happens here.

In a multiple FS environment, given nodes A, B, C, it is arbitrary per
filesystem whether severing node A from the cluster results in node
B or node C becoming the new leader. For each additional filesystem
deployed, the chance that the subsequent leader is always the same
rapidly diminishes. For 2 filesystems, the chance that the cluster
goes down is 50%. For 3: 75%, for 4: 87.25% etc.

The authors of the original RAFT protocol published a dissertation
paper outlining this exact problem and propose a "pre-vote" cycle which
approaches the problem from the recovery side. This is impactful and
complex but guards against actual rogue nodes, and I think it's over
the top for this particular situation. [1]

In our specific situation we have a known runaway election cycle
with no votes ever arriving. This situation is very unusual since
normally when there are elections, votes will be arriving. This is
the key distinction between an "election into the void" and one that
isn't. We can use this as a gate to persist the TERM increase.

The gate then prevents a restarted leader from starting with an
inflated TERM and disrupting a healthy cluster. It will rejoin the
cluster as a normal follower instead.

[1] https://web.stanford.edu/~ouster/cgi-bin/papers/OngaroPhD.pdf Ch-4.2.3

Signed-off-by: Auke Kok <auke.kok@versity.com>

kmod/src/quorum.c

@@ -67,7 +67,15 @@
  *
  * It's critical to raft elections that a participant's term not go
  * backwards in time so each mount also uses its quorum block to store
- * the greatest term it has used in messages.
+ * the greatest term it has used in messages.  A candidate only records
+ * an increased term once another member has responded to its election,
+ * though.  A mount that has lost contact increments its term every
+ * election timeout with nobody answering; if it wrote each increase to
+ * its block it would return after a reboot or fence with a wildly
+ * inflated term, ignore the current leader's smaller term, and need-
+ * lessly fence it.  Any member that does respond has already recorded
+ * the term as it adopted it, and a winner records it in its elect event,
+ * so a term that mattered is never lost.
  *
  * The quorum work still runs in the background while the server is
  * running.  The leader quorum work will regularly send heartbeat
@@ -114,6 +122,7 @@ struct quorum_status {
 	int server_event;
 	int vote_for;
 	unsigned long vote_bits;
+	bool term_persisted;
 	ktime_t timeout;
 };
 
@@ -828,10 +837,16 @@ static void scoutfs_quorum_worker(struct work_struct *work)
 			qst.timeout = election_timeout();
 			scoutfs_inc_counter(sb, quorum_send_request);
 
-			/* store our increased term */
-			ret = update_quorum_block(sb, SCOUTFS_QUORUM_EVENT_TERM, qst.term, true);
-			if (ret < 0)
-				goto out;
+			/*
+			 * Don't persist this increased term yet.  A mount that
+			 * has lost all contact increments its term every timeout
+			 * with nobody answerings. Persisting each increase would
+			 * let it return after a reboot or fence with a wildly
+			 * inflated term and fence the current leader.  We record
+			 * term increase when another member responds, which proves
+			 * the term is actively in circulation.
+			 */
+			qst.term_persisted = false;
 		}
 
 		/* candidates count votes in their term */
@@ -842,6 +857,18 @@ static void scoutfs_quorum_worker(struct work_struct *work)
 					     msg.from, qst.term, msg.from);
 			}
 			scoutfs_inc_counter(sb, quorum_recv_vote);
+
+			/*
+			 * Another member answered our election, so the term is
+			 * now real and should be persisted.
+			 */
+			if (!qst.term_persisted) {
+				ret = update_quorum_block(sb, SCOUTFS_QUORUM_EVENT_TERM,
+							  qst.term, true);
+				if (ret < 0)
+					goto out;
+				qst.term_persisted = true;
+			}
 		}
 
 		/*