From fd1df67efa21ad267ea13624033b31dd664f844a Mon Sep 17 00:00:00 2001 From: Auke Kok Date: Tue, 23 Jun 2026 15:07:16 -0700 Subject: [PATCH] Shorten the default TCP_KEEPALIVE_TIMEOUT_MS to 24s In v1.25-49-gf6746275, we adjusted the default TCP keepalive value for clients from 10s to 60s. The motivation for doing so was to allow the system to withstand an - at that time - unknown networking issue that caused intermittent fencing/disconnects. It was later determined that lock recovery stalls were the culprit, and those were largely addressed by ca. v1.26-21-gf0c79966 which fixes the issue at the root of the problem. The increase from 10s to 60s however was not reverted. We are now seeing issues when scoutfs is deployed as multiple filesystem mounts on a cluster where e.g. the server is dropped for a fencing test - causing a non-leader node to take over from the fenced server. It will issue new fence requests for clients that were previously connected in 30 seconds. But with a 60sec TCP timeout value, any client that is still up will wait silently for the fenced server, until the new leader fences them. This leaves the cluster degraded in case of a 3-node quorum. The precise timeline: - 0sec: nmcli down on node0 - 10sec: new election issued, node1 wins, node2 remains follower - 15sec: node0 fenced (quorum old leader fence) - 45sec: node1 fences clients that failed to recover never reached: - 60sec: node2 client reconnect due to TCP timeout In our scoutfs testing we never see this because we just call umount in our testing script unconditionally. In a multi-fs scenario with real fencing this is never done, and so the cluster degrades almost entirely consistently. Drop the value to 24sec. The new value is well below the highest acceptable value: 45sec. We don't want to lower it too much, and we also don't want it to be a multiple of 15/10 to avoid concurrent intervals causing "flapping" or waves of events excluding. 24s fits well between the minimum of 15 and maximum of 45 and isn't a straight factor or multiple of 10, 15, or 30. Test results: on my 3-node 3-fs setup, this 100% recovered all 3 fs's without degrading the cluster. In 1/20 tests I had to re-start 2 nodes (a "flap") in succession to recover. In the rest of the tests, the cluster recovered as expected. Signed-off-by: Auke Kok --- kmod/src/options.c | 2 +- utils/man/scoutfs.5 | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/kmod/src/options.c b/kmod/src/options.c index b7565d76..1ff6e55f 100644 --- a/kmod/src/options.c +++ b/kmod/src/options.c @@ -136,7 +136,7 @@ static void free_options(struct scoutfs_mount_options *opts) #define MIN_DATA_PREALLOC_BLOCKS 1ULL #define MAX_DATA_PREALLOC_BLOCKS ((unsigned long long)SCOUTFS_BLOCK_SM_MAX) -#define DEFAULT_TCP_KEEPALIVE_TIMEOUT_MS (60 * MSEC_PER_SEC) +#define DEFAULT_TCP_KEEPALIVE_TIMEOUT_MS (24 * MSEC_PER_SEC) static void init_default_options(struct scoutfs_mount_options *opts) { diff --git a/utils/man/scoutfs.5 b/utils/man/scoutfs.5 index d3012419..7efd8006 100644 --- a/utils/man/scoutfs.5 +++ b/utils/man/scoutfs.5 @@ -161,7 +161,7 @@ connection will wait for active TCP packets, before deciding that the connection is dead. This setting is per-mount and only changes the behavior of that mount. .sp -The default value of this setting is 60000msec (60s). Any precision +The default value of this setting is 24000msec (24s). Any precision beyond a whole second is likely unrealistic due to the nature of TCP keepalive mechanisms in the Linux kernel. Valid values are any value higher than 3000 (3s). @@ -170,7 +170,8 @@ The TCP keepalive mechanism is complex and observing a lost connection quickly is important to maintain cluster stability. If the local network suffers from intermittent outages this option may provide some respite to overcome these outages without the cluster becoming -desynchronized. +desynchronized. Values above 30s are discouraged due to the negative +effect on recovery from fencing. .SH VOLUME OPTIONS Volume options are persistent options which are stored in the super block in the metadata device and which apply to all mounts of the volume.