During forced unmount lock calls only return errors today if they get an
immediate error from sending a lock request. If a lock request is
pending then blocked lock calls can hang.
They were meant to return errors after getting error responses that are
generated as the networking client is disconnected, but the response
handler just returns an error itself. Without the response message it
doesn't know which lock key experienced the error. Either associating
requests with lock pointers or passing the initial request data payload
to the response handler both have lifetime implications that make me a
bit nervous.
This instead has lock calls specifically return errors when we're
forcing an unmount. This could be torn out if we took the time to
handle error responses in the lock grant requests.
Signed-off-by: Zach Brown <zab@versity.com>
Remove scoutfs_lock_unmount_begin() which wasn't doing anything.
This is an artifact of back when the lock invalidation in lock.c used to
prune the dcache directly. That can't be done after unmount gets far
enough because the fs core destroys dentries without lock coverage. So
this _unmount begin was flushing invalidation and pruning work before
unmount carried on.
These days this is accomplished by _lock_flush_invalidate() and
_inode_flush_iput().
Signed-off-by: Zach Brown <zab@versity.com>
Previously the only way that pending invalidations would block lock
attempts was by invalidation processing being able to set the new mode
of the lock. New lock attempts ignored pending invalidations and could
effectively postpone invalidation processing indefinitely by unlocking
and locking fast enough.
This results in visible hangs where a remote lock attempt to inspect an
inode can be blocked indefinitely by active use of the inode.
The basic fix of having lock attempts wait when there are pending
invalidations is simple. But we have to be careful not to cause
deadlocks so we only make lock attempts wait when the lock has no other
users. This leaves holes where many active users can still delay
invalidation but fixing that would require tracking lock ownership.
Signed-off-by: Zach Brown <zab@versity.com>
Whole file truncate operations are relatively unique in that they can do
a large amount of work under one cluster lock acquisition. Typically a
large amount of work is done with many reasonably large operations
(write, read, etc).
When truncating a huge amount of extents under one cluster lock other
users of the cluster lock across the network can be starved. While not
strictly incorrect, it makes users like ls seemingly hang and is not a
useful experience.
This makes final inode deletion and releasing modify a file's extents
under multiple cluster lock acquisitions. In both cases we only cycle
the cluster lock if we see a pending invalidation after we've done a
reasonable amount of work.
In the release case this changes behavior by allowing large release
calls to interleave. This is deemed acceptable because, in practice,
userspace archive managers are strictly controlling how tasks release
inodes and aren't relying on the ioctl to resolve concurrency.
Signed-off-by: Zach Brown <zab@versity.com>
This makes yum/dnf upgrade behavior work as expected even if migrating
from previous packaging formats. Previously this would not obviously
'conflict' with the kmod-scoutfs package; it is now marked for
obsolescence.
Signed-off-by: Greg Cymbalski <greg.cymbalski@versity.com>
We can get ENOBUFS when trying to allocate an extent if we've
exhausted our local supply of extents. Rather than returning this
error all the way back to user space, do what the pre-iomap write
paths did and drop our inode lock, release the transaction, and retry.
This gives the allocator an opportunity to refresh its extent pool.
Signed-off-by: Chris Kirby <ckirby@versity.com>
Create the force-enobufs test and use the trigger to inject ENOBUFS
to test its handling in the direct and buffered write paths.
Signed-off-by: Chris Kirby <ckirby@versity.com>
One of the first steps in starting a new server is to fence the
previous server if it's still running. This happens nearly immediately,
and it can take some time for the old server to shut down cleanly.
So we're often fencing when we should really just be patient.
Add a grace period, currently up to about 10s, watching for an old
server to write out its STOP record. After that, go ahead and fence.
Signed-off-by: Chris Kirby <ckirby@versity.com>
Add the server_data_fill_target mount option. This replaces the
previous static target size of the data allocator that the server fills
for client transactions. We calculate the previously static server and
client low water marks based on the option.
This first step doesn't change the default value while making it tunable
only to minimize risk.
Signed-off-by: Zach Brown <zab@versity.com>
The iomap changes removed the call to scoutfs_writepages_sync_none()
from the buffered write path. This was resulting in write stalls due
to an excessive number of dirty pages piling up.
Add that call back to continue enjoying its write smoothing effect.
Signed-off-by: Chris Kirby <ckirby@versity.com>
This is needed for repositories to host legacy/non-per-kver kmod
RPMs for each minor version in a major-only repository, which is
required for various hosted repository providers. This is intended
to be a 'legacy' option for users of the previous scheme.
For non-kver packages, the kmod package now includes the release it
was build for as part of the package name; there is a corresponding
requirement on actually running that release for dependency
resolution to work properly. A 'virtual' package matching the old
naming scheme is provided for ease of use. Dependency resolution
works as expected, even across OS minor version upgrades (provided
that RPMs are provided in a repository).
Signed-off-by: Greg Cymbalski <greg.cymbalski@versity.com>
Allow building the kmod pinned to an exact kver by defining
%{per_kver}=1 at build time:
- Package names use conventions used elsewhere in RHEL to indicate the
kernel version built against
- A virtual package, kmod-scoutfs, preserves the existing name and also
makes it simple to install with yum via repository
- Strict requirements on the kver built against in accordance with
conventions
- Disabling automatic use of weak-modules
If not defining per_kver to 1, legacy behavior is preserved, meaning we
can deliver packages for the same scoutfs commit in both styles
depending on customer needs.
Signed-off-by: Greg Cymbalski <greg.cymbalski@versity.com>
A connecting client must send a greeting message so the server can get
their RID and start processing.
The server acted on connections that hadn't received a greeting with the
RID. It would print messages and would try to fence these connections'
client if it disconnected. The server never received the RID, though,
so there was nothing to fence. The server would shut down if either
fencing raised an error or when the net stack called its notify_down
callback with a 0 RID.
And all of this is actively triggered in the field by port scans that
send either nothing or garbage down connections to the ports that
they're sweeping.
We fix all this by ignoring the connections until we receive a RID. We
don't print the accepted or closed messages, free the connection if it
disconnects, and don't call the notify up or down methods.
Signed-off-by: Zach Brown <zab@versity.com>
This was causing some interesting issues because the vfs code
makes decisions based on the existence of the method while the
read/write iterators ignore the IOCB_DIRECT flag.
Signed-off-by: Chris Kirby <ckirby@versity.com>
scoutfs_readahead() will acquire the cluster read lock if the task
doesn't already hold it. But it doesn't set the corresponding
per-task variable, so the scoutfs_get_block() call won't find
it and mistakenly warns that we aren't holding the lock.
We need to set the per-task variable in scoutfs_readahead(). And since
we can get into this path from scoutfs_fadvise(), and that function
acquires the extent_sem, we also need to first acquire the cluster
read lock and set the per-task variable there too. This fixes a
lock ordering issue that we had not yet seen.
Signed-off-by: Chris Kirby <ckirby@versity.com>
In scoutfs_fallocate() and scoutfs_data_truncate_items(), the
extent_sem and inode index lock were acquired in reverse order
from other code paths. Swap that so everything is consistent.
The ordering should now be as follows:
VFS inode_lock
cluster lock
inode index lock
extent_sem
Signed-off-by: Chris Kirby <ckirby@versity.com>
Switch many code paths to use iomap instead of get_block. This includes
buffered reads and writes, llseek, fiemap, and new support for direct
I/O.
Add KC_USE_IOMAP_FOR_IO and only build the iomap R/W pieces
for EL9.5 and higher. We support iomap for fiemap and llseek
on EL8 and EL9.4.
Signed-off-by: Chris Kirby <ckirby@versity.com>
Reproduce the freed-list commit deadlock and show a fixed server recovers.
The alloc_fill_freed_list trigger stuffs both server_meta_freed heads
to near-full in a single commit. It claims runs of free blocks and
appends them straight into the head blocks, leaking whatever isn't
used.
Filling both heads in one commit makes the wedge reproducible. There's
no window for the drain to empty one head before the other fills. An
unfixed server is left with both heads full and wedges on the next
commit. A fixed server drains them and stays live.
Signed-off-by: Auke Kok <auke.kok@versity.com>
The server commit deadlocks when its freed allocator list head block
fills to near capacity. Both gates that decide whether a transaction has
room measure it as free slots in the clean freed head block:
- hold_commit() admits a holder only if scoutfs_alloc_meta_remaining()
reports enough freed room (2 * COMMIT_HOLD_ALLOC_BUDGET slots).
- empty_list()/fill_list()'s list_has_blocks() proceed only if the head
has extent_mod_blocks() slots.
The clean head is not what a transaction gets: the first dirtying
allocation runs dirty_alloc_blocks(), which rotates in a fresh head block
when the current head is under EMPTY_FREED_THRESH. A clean, nearly-full
head has a full block's worth of room as soon as it's touched. With the
gates refusing on the clean full head, no holder is admitted and the
drains never start, so the rotation in dirty_alloc_blocks() is never
reached. The server spins applying empty commits and the filesystem can't
mount or recover (observed at ~11k empty commits/sec; freed head first_nr
8148 of an 8184 capacity).
Fix the accounting in scoutfs_alloc_meta_remaining(): when the freed list
isn't dirtied yet and the clean head is under EMPTY_FREED_THRESH, report
the room the pending rotation will give, SCOUTFS_ALLOC_LIST_MAX_BLOCKS - 2
(a fresh block, less the old avail and freed head blocks the rotation
frees into it). list_has_blocks() routes through the same function so
fill_list()/empty_list() use identical accounting; otherwise an
avail-low, freed-full commit could still wedge because fill_list()
couldn't refill avail past the clean full freed head.
hold_commit() then admits a holder (or a drain starts) and the first
allocation rotates the full head. The avail gate and the meta_low()
loop-stop are left conservative, so genuine ENOSPC still fails and freeing
loops still commit before overflowing a head.
Signed-off-by: Auke Kok <auke.kok@versity.com>
A failed mount tears down with scoutfs_put_super(), which calls
scoutfs_srch_destroy() first. srch_destroy() does cancel_work_sync() on
the srch compact worker, but that worker can be parked in an
uninterruptible scoutfs_net_sync_request() to a server that will never
respond (e.g. the server is stuck in recovery). Nothing completes the
request: the forced-unmount drain in the net shutdown path only runs for
umount -f, and a failed mount never calls ->umount_begin, so the request
sits on the resend queue and cancel_work_sync() waits forever.
The result is an unkillable D-state mount that survives the SIGKILL a
mount timeout sends, and only clears on reboot:
systemd[1]: data-archive.mount: Killing process 15740 (mount) with signal SIGKILL.
systemd[1]: data-archive.mount: Mount process still around after SIGKILL. Ignoring.
cat /proc/26717/stack
[<0>] __flush_work+0x16f/0x240
[<0>] __cancel_work_sync+0x135/0x1a0
[<0>] scoutfs_srch_destroy+0x33/0x70 [scoutfs]
[<0>] scoutfs_put_super+0x4f/0x1a0 [scoutfs]
[<0>] scoutfs_fill_super+0x260/0x520 [scoutfs]
[<0>] mount_bdev+0xf9/0x150
[<0>] do_new_mount+0x17a/0x310
[<0>] __x64_sys_mount+0x107/0x140
The worker it waits on, blocked in the sync request that never returns:
task:kworker/u269:1 state:D
Workqueue: scoutfs_srch_compact scoutfs_srch_compact_worker [scoutfs]
Call Trace:
__wait_for_common+0x90/0x1d0
scoutfs_net_sync_request+0xdb/0xf0 [scoutfs]
scoutfs_client_srch_get_compact+0x2e/0x40 [scoutfs]
scoutfs_srch_compact_worker+0x64/0x3d0 [scoutfs]
On the fill_super error path, mark forced_unmount and shut the client
connection down before teardown. That drains pending requests with
-ECONNABORTED so the worker returns and srch_destroy()'s cancel_work_sync
completes. It is done for any failure, before the direct put_super call
and before returning to generic_shutdown_super (which calls put_super
when s_root was set), so both teardown paths are covered. sbi is
allocated before any goto out, and scoutfs_client_net_shutdown() is a
no-op when the client or connection was never set up, so early failures
are safe.
Signed-off-by: Auke Kok <auke.kok@versity.com>
A node only needs to be fenced once, but scoutfs_fence_start() can be
called for the same rid more than once. When a new leader starts it
fences the previous leader as it removes it from the quorum
(quorum_block_leader), and that same rid can also be a mounted client
that then fails to recover within the timeout (client_recovery).
The second fence call collides on that name, sysfs returns -EEXIST,
and the error is propagated to fence_pending_recov_worker() which
treats any error as fatal and shuts the server down. On the next
mount a new leader hits the same stale set and the same collision,
so the filesystem can never finish recovery.
Jun 15 09:22:35 kernel: scoutfs f.000000.r.222222: fencing previous leader f.000000.r.111111 at term 183942 in slot 3 with address x.x.x.x:6000
Jun 15 09:22:36 scoutfs-fenced[9194]: [2026-06-15 09:22:36.588037842] server f.000000.r.222222 fencing rid 1111111111111111 at IP x.x.x.x for quorum_block_leader
Jun 15 09:23:09 kernel: scoutfs f.000000.r.222222 error: 30000 ms recovery timeout expired for client rid 1111111111111111, fencing
Jun 15 09:23:09 kernel: sysfs: cannot create duplicate filename '/fs/scoutfs/f.000000.r.222222/fence/1111111111111111'
Jun 15 09:23:09 kernel: scoutfs f.000000.r.222222 error: fence returned err -17, shutting down server
Check the list for the rid and skip the duplicate before creating
sysfs. A pending fence can be freed once it is on the list, so a new
fi->mutex serializes creation against the freeing path: the duplicate
check, sysfs create, and list insert run as a unit, and a fence becomes
visible on the list only once it is fully built. scoutfs_fence_free()
and scoutfs_fence_stop() take the same mutex around removing a fence
and tearing it down, and scoutfs_fence_destroy() drains through
fence_stop() rather than walking the list unlocked.
Signed-off-by: Auke Kok <auke.kok@versity.com>
In v1.25-49-gf6746275, we adjusted the default TCP keepalive
value for clients from 10s to 60s. The motivation for doing so
was to allow the system to withstand an - at that time - unknown
networking issue that caused intermittent fencing/disconnects.
It was later determined that lock recovery stalls were the culprit,
and those were largely addressed by ca. v1.26-21-gf0c79966 which
fixes the issue at the root of the problem. The increase from
10s to 60s however was not reverted.
We are now seeing issues when scoutfs is deployed as multiple
filesystem mounts on a cluster where e.g. the server is dropped
for a fencing test - causing a non-leader node to take over from
the fenced server. It will issue new fence requests for clients
that were previously connected in 30 seconds. But with a 60sec
TCP timeout value, any client that is still up will wait silently
for the fenced server, until the new leader fences them. This leaves
the cluster degraded in case of a 3-node quorum.
The precise timeline:
- 0sec: nmcli down on node0
- 10sec: new election issued, node1 wins, node2 remains follower
- 15sec: node0 fenced (quorum old leader fence)
- 45sec: node1 fences clients that failed to recover
never reached:
- 60sec: node2 client reconnect due to TCP timeout
In our scoutfs testing we never see this because we just call umount
in our testing script unconditionally. In a multi-fs scenario with
real fencing this is never done, and so the cluster degrades almost
entirely consistently.
Drop the value to 24sec. The new value is well below the highest
acceptable value: 45sec. We don't want to lower it too much, and we
also don't want it to be a multiple of 15/10 to avoid concurrent
intervals causing "flapping" or waves of events excluding. 24s fits
well between the minimum of 15 and maximum of 45 and isn't a straight
factor or multiple of 10, 15, or 30.
Test results: on my 3-node 3-fs setup, this 100% recovered all 3
fs's without degrading the cluster. In 1/20 tests I had to re-start
2 nodes (a "flap") in succession to recover. In the rest of the tests,
the cluster recovered as expected.
Signed-off-by: Auke Kok <auke.kok@versity.com>
RHEL7 was the only conditional user of this define, but since
support for that is removed, these can be dropped.
Signed-off-by: Auke Kok <auke.kok@versity.com>
scoutfs_rename2 was a thin shim that validated flags and forwarded
to scoutfs_rename_common. It existed because the old el7
RHEL_IOPS_WRAPPER path used a non-flag-taking .rename op alongside
.rename2; with that path gone there is only one rename method,
and the wrapper has no purpose.
Move the RENAME_NOREPLACE flag validation into scoutfs_rename_common
and point the directory inode_operations .rename slot at it directly.
The symlink inode_operations already used scoutfs_rename_common, so
this also makes symlink rename consistently reject unknown flags
instead of silently accepting them.
Signed-off-by: Auke Kok <auke.kok@versity.com>
el8 already provides stack_trace_save() and stack_trace_print()
in linux/stacktrace.h, so the legacy save_stack_trace/print_stack_trace
fallback inlines are dead. Drop the detection stanza and the inlines.
Signed-off-by: Auke Kok <auke.kok@versity.com>
el8 already provides vm_fault_t and vmf_error(), so the fallback
typedef and inline are dead. Drop the detection stanza and the
two function-signature ifdefs in data.c that switched between the
pre-v4.11 and modern fault handler prototypes.
Signed-off-by: Auke Kok <auke.kok@versity.com>
We still need to keep KC_GENERIC_PERFORM_WRITE_KIOCB_IOV_ITER for
el8. So kc_generic_perform_write remains in place for now.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Zach BrownandGitHub
Greg Cymbalski
Chris Kirby
Auke Kok