Scoutfs print segfaults walking a log_merge btree that has more than one
level. print_btree_block() prints parent (level > 0) items via
print_block_ref(), which invokes the item callback with a NULL value to
print the key portion before printing the child ref:
func(key, 0, 0, NULL, 0, arg);
print_log_merge_item immediately casts val and reads a field,
dereferencing NULL. A log_merge of a single leaf block (height 1) never
hits the parent path; one with height > 1 crashes on the first parent
item:
scoutfs[22043]: segfault at 8 ip 0000000000408ef0 sp 00007fffc5edd8b0 error 4
#0 0x0000000000408ef0 in print_log_merge_item ()
#1 0x000000000040958d in print_btree_block.constprop.0.isra ()
#2 0x000000000040a471 in print_cmd ()
#3 0x0000000000404264 in cmd_execute ()
#4 0x00000000004025c9 in main ()
print_mounted_client_entry has the same bug: it casts val and reads
mcv->addr / mcv->flags with no NULL guard, so a mounted_clients btree of
height > 1 segfaults the same way. print_srch_root_item guards NULL but
casts to scoutfs_srch_compact or scoutfs_srch_file without bounds
checking, so a short or malformed item reads past its end.
Fix all three: return early when val is NULL (printing just the key for
the parent ref where applicable), and bounds-check val_len before each
cast so a short item is reported instead of read past its end.
Signed-off-by: Auke Kok <auke.kok@versity.com>
In v1.25-49-gf6746275, we adjusted the default TCP keepalive
value for clients from 10s to 60s. The motivation for doing so
was to allow the system to withstand an - at that time - unknown
networking issue that caused intermittent fencing/disconnects.
It was later determined that lock recovery stalls were the culprit,
and those were largely addressed by ca. v1.26-21-gf0c79966 which
fixes the issue at the root of the problem. The increase from
10s to 60s however was not reverted.
We are now seeing issues when scoutfs is deployed as multiple
filesystem mounts on a cluster where e.g. the server is dropped
for a fencing test - causing a non-leader node to take over from
the fenced server. It will issue new fence requests for clients
that were previously connected in 30 seconds. But with a 60sec
TCP timeout value, any client that is still up will wait silently
for the fenced server, until the new leader fences them. This leaves
the cluster degraded in case of a 3-node quorum.
The precise timeline:
- 0sec: nmcli down on node0
- 10sec: new election issued, node1 wins, node2 remains follower
- 15sec: node0 fenced (quorum old leader fence)
- 45sec: node1 fences clients that failed to recover
never reached:
- 60sec: node2 client reconnect due to TCP timeout
In our scoutfs testing we never see this because we just call umount
in our testing script unconditionally. In a multi-fs scenario with
real fencing this is never done, and so the cluster degrades almost
entirely consistently.
Drop the value to 24sec. The new value is well below the highest
acceptable value: 45sec. We don't want to lower it too much, and we
also don't want it to be a multiple of 15/10 to avoid concurrent
intervals causing "flapping" or waves of events excluding. 24s fits
well between the minimum of 15 and maximum of 45 and isn't a straight
factor or multiple of 10, 15, or 30.
Test results: on my 3-node 3-fs setup, this 100% recovered all 3
fs's without degrading the cluster. In 1/20 tests I had to re-start
2 nodes (a "flap") in succession to recover. In the rest of the tests,
the cluster recovered as expected.
Signed-off-by: Auke Kok <auke.kok@versity.com>
RHEL7 was the only conditional user of this define, but since
support for that is removed, these can be dropped.
Signed-off-by: Auke Kok <auke.kok@versity.com>
scoutfs_rename2 was a thin shim that validated flags and forwarded
to scoutfs_rename_common. It existed because the old el7
RHEL_IOPS_WRAPPER path used a non-flag-taking .rename op alongside
.rename2; with that path gone there is only one rename method,
and the wrapper has no purpose.
Move the RENAME_NOREPLACE flag validation into scoutfs_rename_common
and point the directory inode_operations .rename slot at it directly.
The symlink inode_operations already used scoutfs_rename_common, so
this also makes symlink rename consistently reject unknown flags
instead of silently accepting them.
Signed-off-by: Auke Kok <auke.kok@versity.com>
el8 already provides stack_trace_save() and stack_trace_print()
in linux/stacktrace.h, so the legacy save_stack_trace/print_stack_trace
fallback inlines are dead. Drop the detection stanza and the inlines.
Signed-off-by: Auke Kok <auke.kok@versity.com>
el8 already provides vm_fault_t and vmf_error(), so the fallback
typedef and inline are dead. Drop the detection stanza and the
two function-signature ifdefs in data.c that switched between the
pre-v4.11 and modern fault handler prototypes.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Only el7 was capable of testing this formatversion. And there
no longer is el7 support. Remove the test.
Signed-off-by: Auke Kok <auke.kok@versity.com>
We still need to keep KC_GENERIC_PERFORM_WRITE_KIOCB_IOV_ITER for
el8. So kc_generic_perform_write remains in place for now.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This is a pre-el7 remnant, possibly from a really old rhel9
version, and was already effectively a stub.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This depends on nfs-utils being installed on the host. Without it
it will skip, and count as a failure. It starts nfs-server and
does a bare exportfs.
- Tests basic read/write/stage/release/data wait.
- Tests setfacl/getfacl.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Scoutfs has supported posix ACLs through the xattr handler table,
which allowed NFS to fetch them through this sideband, which worked
for older kernels.
With recent changes we've pulled in .get_acl because the mainline
kernel is changing how ACL ops are called. But we still left .set_acl
unreachable. This meant that on el9.7 nfs clients could now reach
.get_acl, but still not set them.
With this change, we're finally exposing .set_acl consistently
across all el releases and allowing nfs clients to both get and set
posix ACLs.
Signed-off-by: Auke Kok <auke.kok@versity.com>
El9.8 backported the upstream v6.15.* rename of from_timer to
timer_container_of. Switch the two callers in fence.c and recov.c
to the new style and add a simple kcompat define for older kernels.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Ramps up data preallocation based on the number of online
blocks. This results in a simple 2<<n block allocation pattern
until n=11 (2048) - the default value of data_prealloc_blocks.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Auke Kok
Zach BrownandGitHub