A server that is severed from the rest of the cluster will notice
that it's peers are no longer present. When it does, it will fall
back to becoming a follower, and attempt to rebuild quorum.
It does so by calling elections. Into the void. None of the elections
will receive answers. Each election cycle lasts sub-seconds, so the
TERM is durably increased many times in the span of seconds.
The remainder of the cluster will reform quorum and after about 30s,
fence the original leader. In many cases, the old leader will have
started these futile elections already before the fence completely
shuts down the original server.
When the server is rebooted, it will read it's quorum blocks
and continue off with the inflated TERM value. This causes it to
immediately become the new leader of the cluster, because it's TERM
is greatly beyond the other nodes, which just recovered quorum with
a single TERM increase.
In a single FS environment, this is all fine with a 3-node cluster. At
no point in time is the cluster degraded, but, there is an unnecessary
fence in this sequence: the second fence disrupts a healthy cluster.
Recovery from this second fence is normal. No further TERM inflation
happens here.
In a multiple FS environment, given nodes A, B, C, it is arbitrary per
filesystem whether severing node A from the cluster results in node
B or node C becoming the new leader. For each additional filesystem
deployed, the chance that the subsequent leader is always the same
rapidly diminishes. For 2 filesystems, the chance that the cluster
goes down is 50%. For 3: 75%, for 4: 87.25% etc.
The authors of the original RAFT protocol published a dissertation
paper outlining this exact problem and propose a "pre-vote" cycle which
approaches the problem from the recovery side. This is impactful and
complex but guards against actual rogue nodes, and I think it's over
the top for this particular situation. [1]
In our specific situation we have a known runaway election cycle
with no votes ever arriving. This situation is very unusual since
normally when there are elections, votes will be arriving. This is
the key distinction between an "election into the void" and one that
isn't. We can use this as a gate to persist the TERM increase.
The gate then prevents a restarted leader from starting with an
inflated TERM and disrupting a healthy cluster. It will rejoin the
cluster as a normal follower instead.
[1] https://web.stanford.edu/~ouster/cgi-bin/papers/OngaroPhD.pdf Ch-4.2.3
Signed-off-by: Auke Kok <auke.kok@versity.com>
In v1.25-49-gf6746275, we adjusted the default TCP keepalive
value for clients from 10s to 60s. The motivation for doing so
was to allow the system to withstand an - at that time - unknown
networking issue that caused intermittent fencing/disconnects.
It was later determined that lock recovery stalls were the culprit,
and those were largely addressed by ca. v1.26-21-gf0c79966 which
fixes the issue at the root of the problem. The increase from
10s to 60s however was not reverted.
We are now seeing issues when scoutfs is deployed as multiple
filesystem mounts on a cluster where e.g. the server is dropped
for a fencing test - causing a non-leader node to take over from
the fenced server. It will issue new fence requests for clients
that were previously connected in 30 seconds. But with a 60sec
TCP timeout value, any client that is still up will wait silently
for the fenced server, until the new leader fences them. This leaves
the cluster degraded in case of a 3-node quorum.
The precise timeline:
- 0sec: nmcli down on node0
- 10sec: new election issued, node1 wins, node2 remains follower
- 15sec: node0 fenced (quorum old leader fence)
- 45sec: node1 fences clients that failed to recover
never reached:
- 60sec: node2 client reconnect due to TCP timeout
In our scoutfs testing we never see this because we just call umount
in our testing script unconditionally. In a multi-fs scenario with
real fencing this is never done, and so the cluster degrades almost
entirely consistently.
Drop the value to 24sec. The new value is well below the highest
acceptable value: 45sec. We don't want to lower it too much, and we
also don't want it to be a multiple of 15/10 to avoid concurrent
intervals causing "flapping" or waves of events excluding. 24s fits
well between the minimum of 15 and maximum of 45 and isn't a straight
factor or multiple of 10, 15, or 30.
Test results: on my 3-node 3-fs setup, this 100% recovered all 3
fs's without degrading the cluster. In 1/20 tests I had to re-start
2 nodes (a "flap") in succession to recover. In the rest of the tests,
the cluster recovered as expected.
Signed-off-by: Auke Kok <auke.kok@versity.com>
RHEL7 was the only conditional user of this define, but since
support for that is removed, these can be dropped.
Signed-off-by: Auke Kok <auke.kok@versity.com>
scoutfs_rename2 was a thin shim that validated flags and forwarded
to scoutfs_rename_common. It existed because the old el7
RHEL_IOPS_WRAPPER path used a non-flag-taking .rename op alongside
.rename2; with that path gone there is only one rename method,
and the wrapper has no purpose.
Move the RENAME_NOREPLACE flag validation into scoutfs_rename_common
and point the directory inode_operations .rename slot at it directly.
The symlink inode_operations already used scoutfs_rename_common, so
this also makes symlink rename consistently reject unknown flags
instead of silently accepting them.
Signed-off-by: Auke Kok <auke.kok@versity.com>
el8 already provides stack_trace_save() and stack_trace_print()
in linux/stacktrace.h, so the legacy save_stack_trace/print_stack_trace
fallback inlines are dead. Drop the detection stanza and the inlines.
Signed-off-by: Auke Kok <auke.kok@versity.com>
el8 already provides vm_fault_t and vmf_error(), so the fallback
typedef and inline are dead. Drop the detection stanza and the
two function-signature ifdefs in data.c that switched between the
pre-v4.11 and modern fault handler prototypes.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Only el7 was capable of testing this formatversion. And there
no longer is el7 support. Remove the test.
Signed-off-by: Auke Kok <auke.kok@versity.com>
We still need to keep KC_GENERIC_PERFORM_WRITE_KIOCB_IOV_ITER for
el8. So kc_generic_perform_write remains in place for now.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This is a pre-el7 remnant, possibly from a really old rhel9
version, and was already effectively a stub.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This depends on nfs-utils being installed on the host. Without it
it will skip, and count as a failure. It starts nfs-server and
does a bare exportfs.
- Tests basic read/write/stage/release/data wait.
- Tests setfacl/getfacl.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Scoutfs has supported posix ACLs through the xattr handler table,
which allowed NFS to fetch them through this sideband, which worked
for older kernels.
With recent changes we've pulled in .get_acl because the mainline
kernel is changing how ACL ops are called. But we still left .set_acl
unreachable. This meant that on el9.7 nfs clients could now reach
.get_acl, but still not set them.
With this change, we're finally exposing .set_acl consistently
across all el releases and allowing nfs clients to both get and set
posix ACLs.
Signed-off-by: Auke Kok <auke.kok@versity.com>
El9.8 backported the upstream v6.15.* rename of from_timer to
timer_container_of. Switch the two callers in fence.c and recov.c
to the new style and add a simple kcompat define for older kernels.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Ramps up data preallocation based on the number of online
blocks. This results in a simple 2<<n block allocation pattern
until n=11 (2048) - the default value of data_prealloc_blocks.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Auke Kok
Zach BrownandGitHub