mirror of
https://github.com/versity/scoutfs.git
synced 2026-07-21 07:22:31 +00:00
Auke Kok
414ec36a84
In v1.25-49-gf6746275, we adjusted the default TCP keepalive value for clients from 10s to 60s. The motivation for doing so was to allow the system to withstand an - at that time - unknown networking issue that caused intermittent fencing/disconnects. It was later determined that lock recovery stalls were the culprit, and those were largely addressed by ca. v1.26-21-gf0c79966 which fixes the issue at the root of the problem. The increase from 10s to 60s however was not reverted. We are now seeing issues when scoutfs is deployed as multiple filesystem mounts on a cluster where e.g. the server is dropped for a fencing test - causing a non-leader node to take over from the fenced server. It will issue new fence requests for clients that were previously connected in 30 seconds. But with a 60sec TCP timeout value, any client that is still up will wait silently for the fenced server, until the new leader fences them. This leaves the cluster degraded in case of a 3-node quorum. The precise timeline: - 0sec: nmcli down on node0 - 10sec: new election issued, node1 wins, node2 remains follower - 15sec: node0 fenced (quorum old leader fence) - 45sec: node1 fences clients that failed to recover never reached: - 60sec: node2 client reconnect due to TCP timeout In our scoutfs testing we never see this because we just call umount in our testing script unconditionally. In a multi-fs scenario with real fencing this is never done, and so the cluster degrades almost entirely consistently. Drop the value to 24sec. The new value is well below the highest acceptable value: 45sec. We don't want to lower it too much, and we also don't want it to be a multiple of 15/10 to avoid concurrent intervals causing "flapping" or waves of events excluding. 24s fits well between the minimum of 15 and maximum of 45 and isn't a straight factor or multiple of 10, 15, or 30. Test results: on my 3-node 3-fs setup, this 100% recovered all 3 fs's without degrading the cluster. In 1/20 tests I had to re-start 2 nodes (a "flap") in succession to recover. In the rest of the tests, the cluster recovered as expected. Signed-off-by: Auke Kok <auke.kok@versity.com>