We've seen deletion processing from large-fragmented-free continue on
into later tests. Let's see if this makes sure that deletion finishes
during this test.
Signed-off-by: Zach Brown <zab@zabbo.net>
During forced unmount lock calls only return errors today if they get an
immediate error from sending a lock request. If a lock request is
pending then blocked lock calls can hang.
They were meant to return errors after getting error responses that are
generated as the networking client is disconnected, but the response
handler just returns an error itself. Without the response message it
doesn't know which lock key experienced the error. Either associating
requests with lock pointers or passing the initial request data payload
to the response handler both have lifetime implications that make me a
bit nervous.
This instead has lock calls specifically return errors when we're
forcing an unmount. This could be torn out if we took the time to
handle error responses in the lock grant requests.
Signed-off-by: Zach Brown <zab@versity.com>
Remove scoutfs_lock_unmount_begin() which wasn't doing anything.
This is an artifact of back when the lock invalidation in lock.c used to
prune the dcache directly. That can't be done after unmount gets far
enough because the fs core destroys dentries without lock coverage. So
this _unmount begin was flushing invalidation and pruning work before
unmount carried on.
These days this is accomplished by _lock_flush_invalidate() and
_inode_flush_iput().
Signed-off-by: Zach Brown <zab@versity.com>
Previously the only way that pending invalidations would block lock
attempts was by invalidation processing being able to set the new mode
of the lock. New lock attempts ignored pending invalidations and could
effectively postpone invalidation processing indefinitely by unlocking
and locking fast enough.
This results in visible hangs where a remote lock attempt to inspect an
inode can be blocked indefinitely by active use of the inode.
The basic fix of having lock attempts wait when there are pending
invalidations is simple. But we have to be careful not to cause
deadlocks so we only make lock attempts wait when the lock has no other
users. This leaves holes where many active users can still delay
invalidation but fixing that would require tracking lock ownership.
Signed-off-by: Zach Brown <zab@versity.com>
Whole file truncate operations are relatively unique in that they can do
a large amount of work under one cluster lock acquisition. Typically a
large amount of work is done with many reasonably large operations
(write, read, etc).
When truncating a huge amount of extents under one cluster lock other
users of the cluster lock across the network can be starved. While not
strictly incorrect, it makes users like ls seemingly hang and is not a
useful experience.
This makes final inode deletion and releasing modify a file's extents
under multiple cluster lock acquisitions. In both cases we only cycle
the cluster lock if we see a pending invalidation after we've done a
reasonable amount of work.
In the release case this changes behavior by allowing large release
calls to interleave. This is deemed acceptable because, in practice,
userspace archive managers are strictly controlling how tasks release
inodes and aren't relying on the ioctl to resolve concurrency.
Signed-off-by: Zach Brown <zab@versity.com>
This makes yum/dnf upgrade behavior work as expected even if migrating
from previous packaging formats. Previously this would not obviously
'conflict' with the kmod-scoutfs package; it is now marked for
obsolescence.
Signed-off-by: Greg Cymbalski <greg.cymbalski@versity.com>
We can get ENOBUFS when trying to allocate an extent if we've
exhausted our local supply of extents. Rather than returning this
error all the way back to user space, do what the pre-iomap write
paths did and drop our inode lock, release the transaction, and retry.
This gives the allocator an opportunity to refresh its extent pool.
Signed-off-by: Chris Kirby <ckirby@versity.com>
Create the force-enobufs test and use the trigger to inject ENOBUFS
to test its handling in the direct and buffered write paths.
Signed-off-by: Chris Kirby <ckirby@versity.com>
One of the first steps in starting a new server is to fence the
previous server if it's still running. This happens nearly immediately,
and it can take some time for the old server to shut down cleanly.
So we're often fencing when we should really just be patient.
Add a grace period, currently up to about 10s, watching for an old
server to write out its STOP record. After that, go ahead and fence.
Signed-off-by: Chris Kirby <ckirby@versity.com>
Add the server_data_fill_target mount option. This replaces the
previous static target size of the data allocator that the server fills
for client transactions. We calculate the previously static server and
client low water marks based on the option.
This first step doesn't change the default value while making it tunable
only to minimize risk.
Signed-off-by: Zach Brown <zab@versity.com>
Currently this test reproduces a scenario where disabling the
quorum port on a follower results in that node trying to have
an election on increasing terms. When the quorum port is reopened,
a new leader is elected and fences the previous one.
Signed-off-by: Chris Kirby <ckirby@versity.com>
In t_fs_is_leader(), check the status of the given arg, not
whatever happens in be in $i.
Fix a typo in the quorum-heartbeat-timeout test.
Signed-off-by: Chris Kirby <ckirby@versity.com>
The iomap changes removed the call to scoutfs_writepages_sync_none()
from the buffered write path. This was resulting in write stalls due
to an excessive number of dirty pages piling up.
Add that call back to continue enjoying its write smoothing effect.
Signed-off-by: Chris Kirby <ckirby@versity.com>
This is needed for repositories to host legacy/non-per-kver kmod
RPMs for each minor version in a major-only repository, which is
required for various hosted repository providers. This is intended
to be a 'legacy' option for users of the previous scheme.
For non-kver packages, the kmod package now includes the release it
was build for as part of the package name; there is a corresponding
requirement on actually running that release for dependency
resolution to work properly. A 'virtual' package matching the old
naming scheme is provided for ease of use. Dependency resolution
works as expected, even across OS minor version upgrades (provided
that RPMs are provided in a repository).
Signed-off-by: Greg Cymbalski <greg.cymbalski@versity.com>
Allow building the kmod pinned to an exact kver by defining
%{per_kver}=1 at build time:
- Package names use conventions used elsewhere in RHEL to indicate the
kernel version built against
- A virtual package, kmod-scoutfs, preserves the existing name and also
makes it simple to install with yum via repository
- Strict requirements on the kver built against in accordance with
conventions
- Disabling automatic use of weak-modules
If not defining per_kver to 1, legacy behavior is preserved, meaning we
can deliver packages for the same scoutfs commit in both styles
depending on customer needs.
Signed-off-by: Greg Cymbalski <greg.cymbalski@versity.com>
A connecting client must send a greeting message so the server can get
their RID and start processing.
The server acted on connections that hadn't received a greeting with the
RID. It would print messages and would try to fence these connections'
client if it disconnected. The server never received the RID, though,
so there was nothing to fence. The server would shut down if either
fencing raised an error or when the net stack called its notify_down
callback with a 0 RID.
And all of this is actively triggered in the field by port scans that
send either nothing or garbage down connections to the ports that
they're sweeping.
We fix all this by ignoring the connections until we receive a RID. We
don't print the accepted or closed messages, free the connection if it
disconnects, and don't call the notify up or down methods.
Signed-off-by: Zach Brown <zab@versity.com>
el8_10 backported an attr version that is noisy in generic/062.
There's a patch on the tag here that silences it.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This was causing some interesting issues because the vfs code
makes decisions based on the existence of the method while the
read/write iterators ignore the IOCB_DIRECT flag.
Signed-off-by: Chris Kirby <ckirby@versity.com>
scoutfs_readahead() will acquire the cluster read lock if the task
doesn't already hold it. But it doesn't set the corresponding
per-task variable, so the scoutfs_get_block() call won't find
it and mistakenly warns that we aren't holding the lock.
We need to set the per-task variable in scoutfs_readahead(). And since
we can get into this path from scoutfs_fadvise(), and that function
acquires the extent_sem, we also need to first acquire the cluster
read lock and set the per-task variable there too. This fixes a
lock ordering issue that we had not yet seen.
Signed-off-by: Chris Kirby <ckirby@versity.com>
The mmap test kicks off a background xfs_io job, then does a
scoutfs stage operation in the foreground and sleeps for one
second. Usually that's enough time for the background job to
complete, but very rarely it's not.
This manifests as data corruption when we try to read too soon.
But we can also see from the jobs command output that there's
still a job running.
Use an explicit wait instead of the "sleep 1".
Signed-off-by: Chris Kirby <ckirby@versity.com>
In scoutfs_fallocate() and scoutfs_data_truncate_items(), the
extent_sem and inode index lock were acquired in reverse order
from other code paths. Swap that so everything is consistent.
The ordering should now be as follows:
VFS inode_lock
cluster lock
inode index lock
extent_sem
Signed-off-by: Chris Kirby <ckirby@versity.com>
Use the same check as we do in Makefile.kernelcompat
(look for copy_page_to_iter_nofault in the kernel uio.h
header file) to determine if we should use the no DIO
version of expected xfstests results.
Signed-off-by: Chris Kirby <ckirby@versity.com>
Switch many code paths to use iomap instead of get_block. This includes
buffered reads and writes, llseek, fiemap, and new support for direct
I/O.
Add KC_USE_IOMAP_FOR_IO and only build the iomap R/W pieces
for EL9.5 and higher. We support iomap for fiemap and llseek
on EL8 and EL9.4.
Signed-off-by: Chris Kirby <ckirby@versity.com>
Reproduce the freed-list commit deadlock and show a fixed server recovers.
The alloc_fill_freed_list trigger stuffs both server_meta_freed heads
to near-full in a single commit. It claims runs of free blocks and
appends them straight into the head blocks, leaking whatever isn't
used.
Filling both heads in one commit makes the wedge reproducible. There's
no window for the drain to empty one head before the other fills. An
unfixed server is left with both heads full and wedges on the next
commit. A fixed server drains them and stays live.
Signed-off-by: Auke Kok <auke.kok@versity.com>
The server commit deadlocks when its freed allocator list head block
fills to near capacity. Both gates that decide whether a transaction has
room measure it as free slots in the clean freed head block:
- hold_commit() admits a holder only if scoutfs_alloc_meta_remaining()
reports enough freed room (2 * COMMIT_HOLD_ALLOC_BUDGET slots).
- empty_list()/fill_list()'s list_has_blocks() proceed only if the head
has extent_mod_blocks() slots.
The clean head is not what a transaction gets: the first dirtying
allocation runs dirty_alloc_blocks(), which rotates in a fresh head block
when the current head is under EMPTY_FREED_THRESH. A clean, nearly-full
head has a full block's worth of room as soon as it's touched. With the
gates refusing on the clean full head, no holder is admitted and the
drains never start, so the rotation in dirty_alloc_blocks() is never
reached. The server spins applying empty commits and the filesystem can't
mount or recover (observed at ~11k empty commits/sec; freed head first_nr
8148 of an 8184 capacity).
Fix the accounting in scoutfs_alloc_meta_remaining(): when the freed list
isn't dirtied yet and the clean head is under EMPTY_FREED_THRESH, report
the room the pending rotation will give, SCOUTFS_ALLOC_LIST_MAX_BLOCKS - 2
(a fresh block, less the old avail and freed head blocks the rotation
frees into it). list_has_blocks() routes through the same function so
fill_list()/empty_list() use identical accounting; otherwise an
avail-low, freed-full commit could still wedge because fill_list()
couldn't refill avail past the clean full freed head.
hold_commit() then admits a holder (or a drain starts) and the first
allocation rotates the full head. The avail gate and the meta_low()
loop-stop are left conservative, so genuine ENOSPC still fails and freeing
loops still commit before overflowing a head.
Signed-off-by: Auke Kok <auke.kok@versity.com>
A failed mount tears down with scoutfs_put_super(), which calls
scoutfs_srch_destroy() first. srch_destroy() does cancel_work_sync() on
the srch compact worker, but that worker can be parked in an
uninterruptible scoutfs_net_sync_request() to a server that will never
respond (e.g. the server is stuck in recovery). Nothing completes the
request: the forced-unmount drain in the net shutdown path only runs for
umount -f, and a failed mount never calls ->umount_begin, so the request
sits on the resend queue and cancel_work_sync() waits forever.
The result is an unkillable D-state mount that survives the SIGKILL a
mount timeout sends, and only clears on reboot:
systemd[1]: data-archive.mount: Killing process 15740 (mount) with signal SIGKILL.
systemd[1]: data-archive.mount: Mount process still around after SIGKILL. Ignoring.
cat /proc/26717/stack
[<0>] __flush_work+0x16f/0x240
[<0>] __cancel_work_sync+0x135/0x1a0
[<0>] scoutfs_srch_destroy+0x33/0x70 [scoutfs]
[<0>] scoutfs_put_super+0x4f/0x1a0 [scoutfs]
[<0>] scoutfs_fill_super+0x260/0x520 [scoutfs]
[<0>] mount_bdev+0xf9/0x150
[<0>] do_new_mount+0x17a/0x310
[<0>] __x64_sys_mount+0x107/0x140
The worker it waits on, blocked in the sync request that never returns:
task:kworker/u269:1 state:D
Workqueue: scoutfs_srch_compact scoutfs_srch_compact_worker [scoutfs]
Call Trace:
__wait_for_common+0x90/0x1d0
scoutfs_net_sync_request+0xdb/0xf0 [scoutfs]
scoutfs_client_srch_get_compact+0x2e/0x40 [scoutfs]
scoutfs_srch_compact_worker+0x64/0x3d0 [scoutfs]
On the fill_super error path, mark forced_unmount and shut the client
connection down before teardown. That drains pending requests with
-ECONNABORTED so the worker returns and srch_destroy()'s cancel_work_sync
completes. It is done for any failure, before the direct put_super call
and before returning to generic_shutdown_super (which calls put_super
when s_root was set), so both teardown paths are covered. sbi is
allocated before any goto out, and scoutfs_client_net_shutdown() is a
no-op when the client or connection was never set up, so early failures
are safe.
Signed-off-by: Auke Kok <auke.kok@versity.com>
A node only needs to be fenced once, but scoutfs_fence_start() can be
called for the same rid more than once. When a new leader starts it
fences the previous leader as it removes it from the quorum
(quorum_block_leader), and that same rid can also be a mounted client
that then fails to recover within the timeout (client_recovery).
The second fence call collides on that name, sysfs returns -EEXIST,
and the error is propagated to fence_pending_recov_worker() which
treats any error as fatal and shuts the server down. On the next
mount a new leader hits the same stale set and the same collision,
so the filesystem can never finish recovery.
Jun 15 09:22:35 kernel: scoutfs f.000000.r.222222: fencing previous leader f.000000.r.111111 at term 183942 in slot 3 with address x.x.x.x:6000
Jun 15 09:22:36 scoutfs-fenced[9194]: [2026-06-15 09:22:36.588037842] server f.000000.r.222222 fencing rid 1111111111111111 at IP x.x.x.x for quorum_block_leader
Jun 15 09:23:09 kernel: scoutfs f.000000.r.222222 error: 30000 ms recovery timeout expired for client rid 1111111111111111, fencing
Jun 15 09:23:09 kernel: sysfs: cannot create duplicate filename '/fs/scoutfs/f.000000.r.222222/fence/1111111111111111'
Jun 15 09:23:09 kernel: scoutfs f.000000.r.222222 error: fence returned err -17, shutting down server
Check the list for the rid and skip the duplicate before creating
sysfs. A pending fence can be freed once it is on the list, so a new
fi->mutex serializes creation against the freeing path: the duplicate
check, sysfs create, and list insert run as a unit, and a fence becomes
visible on the list only once it is fully built. scoutfs_fence_free()
and scoutfs_fence_stop() take the same mutex around removing a fence
and tearing it down, and scoutfs_fence_destroy() drains through
fence_stop() rather than walking the list unlocked.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Scoutfs print segfaults walking a log_merge btree that has more than one
level. print_btree_block() prints parent (level > 0) items via
print_block_ref(), which invokes the item callback with a NULL value to
print the key portion before printing the child ref:
func(key, 0, 0, NULL, 0, arg);
print_log_merge_item immediately casts val and reads a field,
dereferencing NULL. A log_merge of a single leaf block (height 1) never
hits the parent path; one with height > 1 crashes on the first parent
item:
scoutfs[22043]: segfault at 8 ip 0000000000408ef0 sp 00007fffc5edd8b0 error 4
#0 0x0000000000408ef0 in print_log_merge_item ()
#1 0x000000000040958d in print_btree_block.constprop.0.isra ()
#2 0x000000000040a471 in print_cmd ()
#3 0x0000000000404264 in cmd_execute ()
#4 0x00000000004025c9 in main ()
print_mounted_client_entry has the same bug: it casts val and reads
mcv->addr / mcv->flags with no NULL guard, so a mounted_clients btree of
height > 1 segfaults the same way. print_srch_root_item guards NULL but
casts to scoutfs_srch_compact or scoutfs_srch_file without bounds
checking, so a short or malformed item reads past its end.
Fix all three: return early when val is NULL (printing just the key for
the parent ref where applicable), and bounds-check val_len before each
cast so a short item is reported instead of read past its end.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Zach BrownandGitHub
Greg Cymbalski
Chris Kirby
Auke Kok