mirror of
https://github.com/versity/scoutfs.git
synced 2026-07-20 06:52:20 +00:00
Auke Kok
01cef6e624
A server that is severed from the rest of the cluster will notice that it's peers are no longer present. When it does, it will fall back to becoming a follower, and attempt to rebuild quorum. It does so by calling elections. Into the void. None of the elections will receive answers. Each election cycle lasts sub-seconds, so the TERM is durably increased many times in the span of seconds. The remainder of the cluster will reform quorum and after about 30s, fence the original leader. In many cases, the old leader will have started these futile elections already before the fence completely shuts down the original server. When the server is rebooted, it will read it's quorum blocks and continue off with the inflated TERM value. This causes it to immediately become the new leader of the cluster, because it's TERM is greatly beyond the other nodes, which just recovered quorum with a single TERM increase. In a single FS environment, this is all fine with a 3-node cluster. At no point in time is the cluster degraded, but, there is an unnecessary fence in this sequence: the second fence disrupts a healthy cluster. Recovery from this second fence is normal. No further TERM inflation happens here. In a multiple FS environment, given nodes A, B, C, it is arbitrary per filesystem whether severing node A from the cluster results in node B or node C becoming the new leader. For each additional filesystem deployed, the chance that the subsequent leader is always the same rapidly diminishes. For 2 filesystems, the chance that the cluster goes down is 50%. For 3: 75%, for 4: 87.25% etc. The authors of the original RAFT protocol published a dissertation paper outlining this exact problem and propose a "pre-vote" cycle which approaches the problem from the recovery side. This is impactful and complex but guards against actual rogue nodes, and I think it's over the top for this particular situation. [1] In our specific situation we have a known runaway election cycle with no votes ever arriving. This situation is very unusual since normally when there are elections, votes will be arriving. This is the key distinction between an "election into the void" and one that isn't. We can use this as a gate to persist the TERM increase. The gate then prevents a restarted leader from starting with an inflated TERM and disrupting a healthy cluster. It will rejoin the cluster as a normal follower instead. [1] https://web.stanford.edu/~ouster/cgi-bin/papers/OngaroPhD.pdf Ch-4.2.3 Signed-off-by: Auke Kok <auke.kok@versity.com>
Introduction
scoutfs is a clustered in-kernel Linux filesystem designed to support large archival systems. It features additional interfaces and metadata so that archive agents can perform their maintenance workflows without walking all the files in the namespace. Its cluster support lets deployments add nodes to satisfy archival tier bandwidth targets.
The design goal is to reach file populations in the trillions, with the archival bandwidth to match, while remaining operational and responsive.
Highlights of the design and implementation include:
- Fully consistent POSIX semantics between nodes
- Atomic transactions to maintain consistent persistent structures
- Integrated archival metadata replaces syncing to external databases
- Dynamic seperation of resources lets nodes write in parallel
- 64bit throughout; no limits on file or directory sizes or counts
- Open GPLv2 implementation
Community Mailing List
Please join us on the open scoutfs-devel@scoutfs.org mailing list hosted on Google Groups