We need a few more cherry picked fixes for xfstests that the same things
that we've fixed in the el10 branch: egrep, coreutils mv string changes,
and a new pattern of xfstests probing other filesystem types on our
devices leaking into dmesg. This moves the xfstests tag we're using
by those 3 cherry picks, and adds the filter.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Adds compat handlers for block_write_begin and block_write_end to take
a folio argument instead of page, as this is needed since v6.11-rc1-54-g9f04609f74ec.
To avoid having two duplicate page_mkwrite functions there's now a
complete page/folio ifdef split here to make it handle either. This is
ugly but it's the most straightforward solution here and avoids more
obscure macros.
Signed-off-by: Auke Kok <auke.kok@versity.com>
In el10, we don't need compat for list_lru_walk, as was intended
in 8b6418fb. However, the actual redefine from kc_list_lru_walk to
list_lru_walk was omitted. It wasn't needed until el10. Add it now.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Since v4.6-rc3-29-g6192269444eb there has been a special readdir VFS
method that can be called for the same directory multiple times in
parallel, without any additional VFS locking. The VFS has provided a
WRAP_DIR_ITER() macro to re-wrap the method with extra locking, in case
the method wasn't safe for this.
With el10, the old .readdir method is now gone, and we have no choice
but to either use the wrapper, or just hook up our readdir() method to
the .iterate_shared op.
From what I can see, our implementation is safe to do this.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Based on my reading of the gfs2 driver, it appears it's likely the safer
approach to take copy_splice_read instead of filemap_splice_read as it
may potentially lead to cluster deadlocks.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Due to folios, the kernel will call scoutfs_writepages() and this
becomes unused. It could be ported but the helper function to call isn't
exported anymore.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This works together with the dropped block_write_full_page(), allowing
us to drop the _writepage() method as long as we implement
_writepages(). Since v5.19-rc3-395-g67235182a41c. This used to be the
.migratepage() method.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Since v5.1-rc3-29-gaa30f47cf666, and in el9, there are changes to reduce
the amount of boilerplate code needed to hook up lots of attribute files
using a .default_groups member. In el10, this is the required method as
.default_attrs has been removed. This touches every sysfs part that we
have.
Signed-off-by: Auke Kok <auke.kok@versity.com>
In v6.9-rc4-8-gead083aeeed9, this now takes a struct file argument,
adding to the ifdef salad we've got going on here.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Since ~v6.5-rc1-95-g0d72b92883c6, generic_fillattr() asks us to pass
through the request_mask from the caller. This allows it to only
request a subset.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Yet another major shrinker API evolution in v6.6-rc4-53-gc42d50aefd17.
The struct shrinker now has to be dynamically allocated. This is
purposely a backwards incompatible break.
Collapse the previous KC_ALLOC_SHRINKER, KC_INIT_SHRINKER_FUNCS,
and KC_REGISTER_SHRINKER macros into a single KC_SETUP_SHRINKER
macro. The three operations have to happen in different orders on
different kernel APIs (the name is needed at alloc time on el10
and at register time on KC_SHRINKER_NAME kernels), so coupling
them keeps the ordering correct per kernel.
Add KC_SHRINKER_IS_NULL so callers can detect shrinker_alloc()
failure on el10 and return -ENOMEM. The macro compiles to a
constant 0 on older kernels where the shrinker is an embedded
struct that cannot fail allocation.
Signed-off-by: Auke Kok <auke.kok@versity.com>
The return type always has been int, so, we just need to add return
value checking and do something with it. We could return -ENOMEM here as
well, either way it'll fall all the way through no matter what.
This is since v6.4-rc2-100-g83f2caaaf9cb.
Signed-off-by: Auke Kok <auke.kok@versity.com>
In v6.8-9146-gc759e609030c, the second argument for __assign_str() was
removed, as the second parameter is already derived from the __string()
definition and no longer needed. We have to do a little digging in
headers here to find the definition.
Note the missing `;` at a few places... it has to be added now.
Signed-off-by: Auke Kok <auke.kok@versity.com>
v6.9-rc4-29-g203c1ce0bb06 removes bd_inode. The canonical replacement is
bd_mapping->host, where applicable. We have one use where we directly
need the mapping instead of the inode, as well.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Instead of defining a struct that ends with a flex array member with
`val[0]`, the compiler now balks at this since technically, the spec
considers this unsanitary. As a result however, we can't memcpy to
`struct->val` since that's a pointer and now we're writing something of
a different length (u8's in our case) into something that's of pointer
size. So there we have to do the opposite, and memcpy to
&struct->val[0].
Signed-off-by: Auke Kok <auke.kok@versity.com>
In v6.12-rc1-3-g5f60d5f6bbc1, asm/unaligned.h only included
asm-generic/unaligned.h and that was cleaned up from architecture
specific things. Everyone should now include linux/unaligned.h and the
former include was removed.
A quick peek at server.c shows that while included, it no longer uses
any function from this header at all, so it can just be dropped.
Signed-off-by: Auke Kok <auke.kok@versity.com>
In v6.6-rc5-1-g077c212f0344, one can no longer directly access the
inode m_time and a_time etc. We have to go through these static inline
functions to get to them.
Further back, ctime accessors were added in v6.5-rc1-7-g9b6304c1d537,
and need to be applied as well.
Signed-off-by: Auke Kok <auke.kok@versity.com>
In v6.1-rc5-2-ge9a688bcb193, get_random_u32_below() becomes available and
can start replacing prandom_bytes_max(). Switch to it where we can.
get_random_bytes() has been available since el7, so also replace
prandom_bytes() where we're using it.
Signed-off-by: Auke Kok <auke.kok@versity.com>
In RHEL10, the grep version is bumped from 3.6 to 3.11, and grep
no longer recognizes the \Z character anymore.
We have 2 solutions: We can either choose to use `grep -P` to
continue using it, or, alternatively, we can choose a different
`null` match to have an effectively empty exclude list.
The latter seems easy enough: By default, we can just exclude
empty lines ("^$") obtaining the exact same behavior as before.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This is somewhat cumbersome, we want to see the error message, but the
format changes enough to make this messy. We opt to change the golden to
the new format, which only shows one of the arguments in its error
output: the thing that cannot be overwritten. We then add a filter that
rewrites the old output format with sed patterns to be exactly like the
new format, so this will work everywhere again, without changing or
adding filters to obscure error messages.
Signed-off-by: Auke Kok <auke.kok@versity.com>
The new format in el10 has non-hex output, separated by a comma. Add the
additional filter string so this works as expected.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This was causing some interesting issues because the vfs code
makes decisions based on the existence of the method while the
read/write iterators ignore the IOCB_DIRECT flag.
Signed-off-by: Chris Kirby <ckirby@versity.com>
scoutfs_readahead() will acquire the cluster read lock if the task
doesn't already hold it. But it doesn't set the corresponding
per-task variable, so the scoutfs_get_block() call won't find
it and mistakenly warns that we aren't holding the lock.
We need to set the per-task variable in scoutfs_readahead(). And since
we can get into this path from scoutfs_fadvise(), and that function
acquires the extent_sem, we also need to first acquire the cluster
read lock and set the per-task variable there too. This fixes a
lock ordering issue that we had not yet seen.
Signed-off-by: Chris Kirby <ckirby@versity.com>
The mmap test kicks off a background xfs_io job, then does a
scoutfs stage operation in the foreground and sleeps for one
second. Usually that's enough time for the background job to
complete, but very rarely it's not.
This manifests as data corruption when we try to read too soon.
But we can also see from the jobs command output that there's
still a job running.
Use an explicit wait instead of the "sleep 1".
Signed-off-by: Chris Kirby <ckirby@versity.com>
In scoutfs_fallocate() and scoutfs_data_truncate_items(), the
extent_sem and inode index lock were acquired in reverse order
from other code paths. Swap that so everything is consistent.
The ordering should now be as follows:
VFS inode_lock
cluster lock
inode index lock
extent_sem
Signed-off-by: Chris Kirby <ckirby@versity.com>
Use the same check as we do in Makefile.kernelcompat
(look for copy_page_to_iter_nofault in the kernel uio.h
header file) to determine if we should use the no DIO
version of expected xfstests results.
Signed-off-by: Chris Kirby <ckirby@versity.com>
Switch many code paths to use iomap instead of get_block. This includes
buffered reads and writes, llseek, fiemap, and new support for direct
I/O.
Add KC_USE_IOMAP_FOR_IO and only build the iomap R/W pieces
for EL9.5 and higher. We support iomap for fiemap and llseek
on EL8 and EL9.4.
Signed-off-by: Chris Kirby <ckirby@versity.com>
Reproduce the freed-list commit deadlock and show a fixed server recovers.
The alloc_fill_freed_list trigger stuffs both server_meta_freed heads
to near-full in a single commit. It claims runs of free blocks and
appends them straight into the head blocks, leaking whatever isn't
used.
Filling both heads in one commit makes the wedge reproducible. There's
no window for the drain to empty one head before the other fills. An
unfixed server is left with both heads full and wedges on the next
commit. A fixed server drains them and stays live.
Signed-off-by: Auke Kok <auke.kok@versity.com>
The server commit deadlocks when its freed allocator list head block
fills to near capacity. Both gates that decide whether a transaction has
room measure it as free slots in the clean freed head block:
- hold_commit() admits a holder only if scoutfs_alloc_meta_remaining()
reports enough freed room (2 * COMMIT_HOLD_ALLOC_BUDGET slots).
- empty_list()/fill_list()'s list_has_blocks() proceed only if the head
has extent_mod_blocks() slots.
The clean head is not what a transaction gets: the first dirtying
allocation runs dirty_alloc_blocks(), which rotates in a fresh head block
when the current head is under EMPTY_FREED_THRESH. A clean, nearly-full
head has a full block's worth of room as soon as it's touched. With the
gates refusing on the clean full head, no holder is admitted and the
drains never start, so the rotation in dirty_alloc_blocks() is never
reached. The server spins applying empty commits and the filesystem can't
mount or recover (observed at ~11k empty commits/sec; freed head first_nr
8148 of an 8184 capacity).
Fix the accounting in scoutfs_alloc_meta_remaining(): when the freed list
isn't dirtied yet and the clean head is under EMPTY_FREED_THRESH, report
the room the pending rotation will give, SCOUTFS_ALLOC_LIST_MAX_BLOCKS - 2
(a fresh block, less the old avail and freed head blocks the rotation
frees into it). list_has_blocks() routes through the same function so
fill_list()/empty_list() use identical accounting; otherwise an
avail-low, freed-full commit could still wedge because fill_list()
couldn't refill avail past the clean full freed head.
hold_commit() then admits a holder (or a drain starts) and the first
allocation rotates the full head. The avail gate and the meta_low()
loop-stop are left conservative, so genuine ENOSPC still fails and freeing
loops still commit before overflowing a head.
Signed-off-by: Auke Kok <auke.kok@versity.com>
A failed mount tears down with scoutfs_put_super(), which calls
scoutfs_srch_destroy() first. srch_destroy() does cancel_work_sync() on
the srch compact worker, but that worker can be parked in an
uninterruptible scoutfs_net_sync_request() to a server that will never
respond (e.g. the server is stuck in recovery). Nothing completes the
request: the forced-unmount drain in the net shutdown path only runs for
umount -f, and a failed mount never calls ->umount_begin, so the request
sits on the resend queue and cancel_work_sync() waits forever.
The result is an unkillable D-state mount that survives the SIGKILL a
mount timeout sends, and only clears on reboot:
systemd[1]: data-archive.mount: Killing process 15740 (mount) with signal SIGKILL.
systemd[1]: data-archive.mount: Mount process still around after SIGKILL. Ignoring.
cat /proc/26717/stack
[<0>] __flush_work+0x16f/0x240
[<0>] __cancel_work_sync+0x135/0x1a0
[<0>] scoutfs_srch_destroy+0x33/0x70 [scoutfs]
[<0>] scoutfs_put_super+0x4f/0x1a0 [scoutfs]
[<0>] scoutfs_fill_super+0x260/0x520 [scoutfs]
[<0>] mount_bdev+0xf9/0x150
[<0>] do_new_mount+0x17a/0x310
[<0>] __x64_sys_mount+0x107/0x140
The worker it waits on, blocked in the sync request that never returns:
task:kworker/u269:1 state:D
Workqueue: scoutfs_srch_compact scoutfs_srch_compact_worker [scoutfs]
Call Trace:
__wait_for_common+0x90/0x1d0
scoutfs_net_sync_request+0xdb/0xf0 [scoutfs]
scoutfs_client_srch_get_compact+0x2e/0x40 [scoutfs]
scoutfs_srch_compact_worker+0x64/0x3d0 [scoutfs]
On the fill_super error path, mark forced_unmount and shut the client
connection down before teardown. That drains pending requests with
-ECONNABORTED so the worker returns and srch_destroy()'s cancel_work_sync
completes. It is done for any failure, before the direct put_super call
and before returning to generic_shutdown_super (which calls put_super
when s_root was set), so both teardown paths are covered. sbi is
allocated before any goto out, and scoutfs_client_net_shutdown() is a
no-op when the client or connection was never set up, so early failures
are safe.
Signed-off-by: Auke Kok <auke.kok@versity.com>
A node only needs to be fenced once, but scoutfs_fence_start() can be
called for the same rid more than once. When a new leader starts it
fences the previous leader as it removes it from the quorum
(quorum_block_leader), and that same rid can also be a mounted client
that then fails to recover within the timeout (client_recovery).
The second fence call collides on that name, sysfs returns -EEXIST,
and the error is propagated to fence_pending_recov_worker() which
treats any error as fatal and shuts the server down. On the next
mount a new leader hits the same stale set and the same collision,
so the filesystem can never finish recovery.
Jun 15 09:22:35 kernel: scoutfs f.000000.r.222222: fencing previous leader f.000000.r.111111 at term 183942 in slot 3 with address x.x.x.x:6000
Jun 15 09:22:36 scoutfs-fenced[9194]: [2026-06-15 09:22:36.588037842] server f.000000.r.222222 fencing rid 1111111111111111 at IP x.x.x.x for quorum_block_leader
Jun 15 09:23:09 kernel: scoutfs f.000000.r.222222 error: 30000 ms recovery timeout expired for client rid 1111111111111111, fencing
Jun 15 09:23:09 kernel: sysfs: cannot create duplicate filename '/fs/scoutfs/f.000000.r.222222/fence/1111111111111111'
Jun 15 09:23:09 kernel: scoutfs f.000000.r.222222 error: fence returned err -17, shutting down server
Check the list for the rid and skip the duplicate before creating
sysfs. A pending fence can be freed once it is on the list, so a new
fi->mutex serializes creation against the freeing path: the duplicate
check, sysfs create, and list insert run as a unit, and a fence becomes
visible on the list only once it is fully built. scoutfs_fence_free()
and scoutfs_fence_stop() take the same mutex around removing a fence
and tearing it down, and scoutfs_fence_destroy() drains through
fence_stop() rather than walking the list unlocked.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Scoutfs print segfaults walking a log_merge btree that has more than one
level. print_btree_block() prints parent (level > 0) items via
print_block_ref(), which invokes the item callback with a NULL value to
print the key portion before printing the child ref:
func(key, 0, 0, NULL, 0, arg);
print_log_merge_item immediately casts val and reads a field,
dereferencing NULL. A log_merge of a single leaf block (height 1) never
hits the parent path; one with height > 1 crashes on the first parent
item:
scoutfs[22043]: segfault at 8 ip 0000000000408ef0 sp 00007fffc5edd8b0 error 4
#0 0x0000000000408ef0 in print_log_merge_item ()
#1 0x000000000040958d in print_btree_block.constprop.0.isra ()
#2 0x000000000040a471 in print_cmd ()
#3 0x0000000000404264 in cmd_execute ()
#4 0x00000000004025c9 in main ()
print_mounted_client_entry has the same bug: it casts val and reads
mcv->addr / mcv->flags with no NULL guard, so a mounted_clients btree of
height > 1 segfaults the same way. print_srch_root_item guards NULL but
casts to scoutfs_srch_compact or scoutfs_srch_file without bounds
checking, so a short or malformed item reads past its end.
Fix all three: return early when val is NULL (printing just the key for
the parent ref where applicable), and bounds-check val_len before each
cast so a short item is reported instead of read past its end.
Signed-off-by: Auke Kok <auke.kok@versity.com>
If a set of tests is provided, loop over the entire set the
requested number of times. Start and stop any requested tracing
across the set boundary.
Signed-off-by: Chris Kirby <ckirby@versity.com>
In v1.25-49-gf6746275, we adjusted the default TCP keepalive
value for clients from 10s to 60s. The motivation for doing so
was to allow the system to withstand an - at that time - unknown
networking issue that caused intermittent fencing/disconnects.
It was later determined that lock recovery stalls were the culprit,
and those were largely addressed by ca. v1.26-21-gf0c79966 which
fixes the issue at the root of the problem. The increase from
10s to 60s however was not reverted.
We are now seeing issues when scoutfs is deployed as multiple
filesystem mounts on a cluster where e.g. the server is dropped
for a fencing test - causing a non-leader node to take over from
the fenced server. It will issue new fence requests for clients
that were previously connected in 30 seconds. But with a 60sec
TCP timeout value, any client that is still up will wait silently
for the fenced server, until the new leader fences them. This leaves
the cluster degraded in case of a 3-node quorum.
The precise timeline:
- 0sec: nmcli down on node0
- 10sec: new election issued, node1 wins, node2 remains follower
- 15sec: node0 fenced (quorum old leader fence)
- 45sec: node1 fences clients that failed to recover
never reached:
- 60sec: node2 client reconnect due to TCP timeout
In our scoutfs testing we never see this because we just call umount
in our testing script unconditionally. In a multi-fs scenario with
real fencing this is never done, and so the cluster degrades almost
entirely consistently.
Drop the value to 24sec. The new value is well below the highest
acceptable value: 45sec. We don't want to lower it too much, and we
also don't want it to be a multiple of 15/10 to avoid concurrent
intervals causing "flapping" or waves of events excluding. 24s fits
well between the minimum of 15 and maximum of 45 and isn't a straight
factor or multiple of 10, 15, or 30.
Test results: on my 3-node 3-fs setup, this 100% recovered all 3
fs's without degrading the cluster. In 1/20 tests I had to re-start
2 nodes (a "flap") in succession to recover. In the rest of the tests,
the cluster recovered as expected.
Signed-off-by: Auke Kok <auke.kok@versity.com>
RHEL7 was the only conditional user of this define, but since
support for that is removed, these can be dropped.
Signed-off-by: Auke Kok <auke.kok@versity.com>
scoutfs_rename2 was a thin shim that validated flags and forwarded
to scoutfs_rename_common. It existed because the old el7
RHEL_IOPS_WRAPPER path used a non-flag-taking .rename op alongside
.rename2; with that path gone there is only one rename method,
and the wrapper has no purpose.
Move the RENAME_NOREPLACE flag validation into scoutfs_rename_common
and point the directory inode_operations .rename slot at it directly.
The symlink inode_operations already used scoutfs_rename_common, so
this also makes symlink rename consistently reject unknown flags
instead of silently accepting them.
Signed-off-by: Auke Kok <auke.kok@versity.com>
el8 already provides stack_trace_save() and stack_trace_print()
in linux/stacktrace.h, so the legacy save_stack_trace/print_stack_trace
fallback inlines are dead. Drop the detection stanza and the inlines.
Signed-off-by: Auke Kok <auke.kok@versity.com>
el8 already provides vm_fault_t and vmf_error(), so the fallback
typedef and inline are dead. Drop the detection stanza and the
two function-signature ifdefs in data.c that switched between the
pre-v4.11 and modern fault handler prototypes.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Only el7 was capable of testing this formatversion. And there
no longer is el7 support. Remove the test.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Auke Kok
Zach BrownandGitHub
Chris Kirby