Shorten the default TCP_KEEPALIVE_TIMEOUT_MS to 24s

In v1.25-49-gf6746275, we adjusted the default TCP keepalive
value for clients from 10s to 60s. The motivation for doing so
was to allow the system to withstand an - at that time - unknown
networking issue that caused intermittent fencing/disconnects.

It was later determined that lock recovery stalls were the culprit,
and those were largely addressed by ca. v1.26-21-gf0c79966 which
fixes the issue at the root of the problem. The increase from
10s to 60s however was not reverted.

We are now seeing issues when scoutfs is deployed as multiple
filesystem mounts on a cluster where e.g. the server is dropped
for a fencing test - causing a non-leader node to take over from
the fenced server. It will issue new fence requests for clients
that were previously connected in 30 seconds. But with a 60sec
TCP timeout value, any client that is still up will wait silently
for the fenced server, until the new leader fences them. This leaves
the cluster degraded in case of a 3-node quorum.

The precise timeline:
  -  0sec: nmcli down on node0
  - 10sec: new election issued, node1 wins, node2 remains follower
  - 15sec: node0 fenced (quorum old leader fence)
  - 45sec: node1 fences clients that failed to recover
  never reached:
  - 60sec: node2 client reconnect due to TCP timeout

In our scoutfs testing we never see this because we just call umount
in our testing script unconditionally. In a multi-fs scenario with
real fencing this is never done, and so the cluster degrades almost
entirely consistently.

Drop the value to 24sec. The new value is well below the highest
acceptable value: 45sec. We don't want to lower it too much, and we
also don't want it to be a multiple of 15/10 to avoid concurrent
intervals causing "flapping" or waves of events excluding. 24s fits
well between the minimum of 15 and maximum of 45 and isn't a straight
factor or multiple of 10, 15, or 30.

Test results: on my 3-node 3-fs setup, this 100% recovered all 3
fs's without degrading the cluster. In 1/20 tests I had to re-start
2 nodes (a "flap") in succession to recover. In the rest of the tests,
the cluster recovered as expected.

Signed-off-by: Auke Kok <auke.kok@versity.com>

kmod/src/options.c

@@ -136,7 +136,7 @@ static void free_options(struct scoutfs_mount_options *opts)
 #define MIN_DATA_PREALLOC_BLOCKS	1ULL
 #define MAX_DATA_PREALLOC_BLOCKS	((unsigned long long)SCOUTFS_BLOCK_SM_MAX)
 
-#define DEFAULT_TCP_KEEPALIVE_TIMEOUT_MS	(60 * MSEC_PER_SEC)
+#define DEFAULT_TCP_KEEPALIVE_TIMEOUT_MS	(24 * MSEC_PER_SEC)
 
 static void init_default_options(struct scoutfs_mount_options *opts)
 {

utils/man/scoutfs.5

@@ -161,7 +161,7 @@ connection will wait for active TCP packets, before deciding that
 the connection is dead. This setting is per-mount and only changes
 the behavior of that mount.
 .sp
-The default value of this setting is 60000msec (60s). Any precision
+The default value of this setting is 24000msec (24s). Any precision
 beyond a whole second is likely unrealistic due to the nature of
 TCP keepalive mechanisms in the Linux kernel. Valid values are any
 value higher than 3000 (3s).
@@ -170,7 +170,8 @@ The TCP keepalive mechanism is complex and observing a lost connection
 quickly is important to maintain cluster stability. If the local
 network suffers from intermittent outages this option may provide
 some respite to overcome these outages without the cluster becoming
-desynchronized.
+desynchronized. Values above 30s are discouraged due to the negative
+effect on recovery from fencing.
 .SH VOLUME OPTIONS
 Volume options are persistent options which are stored in the super
 block in the metadata device and which apply to all mounts of the volume.