mirror of
https://github.com/versity/scoutfs.git
synced 2026-07-24 17:12:51 +00:00
Auke Kok
414ec36a84
In v1.25-49-gf6746275, we adjusted the default TCP keepalive value for clients from 10s to 60s. The motivation for doing so was to allow the system to withstand an - at that time - unknown networking issue that caused intermittent fencing/disconnects. It was later determined that lock recovery stalls were the culprit, and those were largely addressed by ca. v1.26-21-gf0c79966 which fixes the issue at the root of the problem. The increase from 10s to 60s however was not reverted. We are now seeing issues when scoutfs is deployed as multiple filesystem mounts on a cluster where e.g. the server is dropped for a fencing test - causing a non-leader node to take over from the fenced server. It will issue new fence requests for clients that were previously connected in 30 seconds. But with a 60sec TCP timeout value, any client that is still up will wait silently for the fenced server, until the new leader fences them. This leaves the cluster degraded in case of a 3-node quorum. The precise timeline: - 0sec: nmcli down on node0 - 10sec: new election issued, node1 wins, node2 remains follower - 15sec: node0 fenced (quorum old leader fence) - 45sec: node1 fences clients that failed to recover never reached: - 60sec: node2 client reconnect due to TCP timeout In our scoutfs testing we never see this because we just call umount in our testing script unconditionally. In a multi-fs scenario with real fencing this is never done, and so the cluster degrades almost entirely consistently. Drop the value to 24sec. The new value is well below the highest acceptable value: 45sec. We don't want to lower it too much, and we also don't want it to be a multiple of 15/10 to avoid concurrent intervals causing "flapping" or waves of events excluding. 24s fits well between the minimum of 15 and maximum of 45 and isn't a straight factor or multiple of 10, 15, or 30. Test results: on my 3-node 3-fs setup, this 100% recovered all 3 fs's without degrading the cluster. In 1/20 tests I had to re-start 2 nodes (a "flap") in succession to recover. In the rest of the tests, the cluster recovered as expected. Signed-off-by: Auke Kok <auke.kok@versity.com>
Introduction
scoutfs is a clustered in-kernel Linux filesystem designed to support large archival systems. It features additional interfaces and metadata so that archive agents can perform their maintenance workflows without walking all the files in the namespace. Its cluster support lets deployments add nodes to satisfy archival tier bandwidth targets.
The design goal is to reach file populations in the trillions, with the archival bandwidth to match, while remaining operational and responsive.
Highlights of the design and implementation include:
- Fully consistent POSIX semantics between nodes
- Atomic transactions to maintain consistent persistent structures
- Integrated archival metadata replaces syncing to external databases
- Dynamic seperation of resources lets nodes write in parallel
- 64bit throughout; no limits on file or directory sizes or counts
- Open GPLv2 implementation
Community Mailing List
Please join us on the open scoutfs-devel@scoutfs.org mailing list hosted on Google Groups