A failed mount tears down with scoutfs_put_super(), which calls
scoutfs_srch_destroy() first. srch_destroy() does cancel_work_sync() on
the srch compact worker, but that worker can be parked in an
uninterruptible scoutfs_net_sync_request() to a server that will never
respond (e.g. the server is stuck in recovery). Nothing completes the
request: the forced-unmount drain in the net shutdown path only runs for
umount -f, and a failed mount never calls ->umount_begin, so the request
sits on the resend queue and cancel_work_sync() waits forever.
The result is an unkillable D-state mount that survives the SIGKILL a
mount timeout sends, and only clears on reboot:
systemd[1]: data-archive.mount: Killing process 15740 (mount) with signal SIGKILL.
systemd[1]: data-archive.mount: Mount process still around after SIGKILL. Ignoring.
cat /proc/26717/stack
[<0>] __flush_work+0x16f/0x240
[<0>] __cancel_work_sync+0x135/0x1a0
[<0>] scoutfs_srch_destroy+0x33/0x70 [scoutfs]
[<0>] scoutfs_put_super+0x4f/0x1a0 [scoutfs]
[<0>] scoutfs_fill_super+0x260/0x520 [scoutfs]
[<0>] mount_bdev+0xf9/0x150
[<0>] do_new_mount+0x17a/0x310
[<0>] __x64_sys_mount+0x107/0x140
The worker it waits on, blocked in the sync request that never returns:
task:kworker/u269:1 state:D
Workqueue: scoutfs_srch_compact scoutfs_srch_compact_worker [scoutfs]
Call Trace:
__wait_for_common+0x90/0x1d0
scoutfs_net_sync_request+0xdb/0xf0 [scoutfs]
scoutfs_client_srch_get_compact+0x2e/0x40 [scoutfs]
scoutfs_srch_compact_worker+0x64/0x3d0 [scoutfs]
On the fill_super error path, mark forced_unmount and shut the client
connection down before teardown. That drains pending requests with
-ECONNABORTED so the worker returns and srch_destroy()'s cancel_work_sync
completes. It is done for any failure, before the direct put_super call
and before returning to generic_shutdown_super (which calls put_super
when s_root was set), so both teardown paths are covered. sbi is
allocated before any goto out, and scoutfs_client_net_shutdown() is a
no-op when the client or connection was never set up, so early failures
are safe.
Signed-off-by: Auke Kok <auke.kok@versity.com>
A node only needs to be fenced once, but scoutfs_fence_start() can be
called for the same rid more than once. When a new leader starts it
fences the previous leader as it removes it from the quorum
(quorum_block_leader), and that same rid can also be a mounted client
that then fails to recover within the timeout (client_recovery).
The second fence call collides on that name, sysfs returns -EEXIST,
and the error is propagated to fence_pending_recov_worker() which
treats any error as fatal and shuts the server down. On the next
mount a new leader hits the same stale set and the same collision,
so the filesystem can never finish recovery.
Jun 15 09:22:35 kernel: scoutfs f.000000.r.222222: fencing previous leader f.000000.r.111111 at term 183942 in slot 3 with address x.x.x.x:6000
Jun 15 09:22:36 scoutfs-fenced[9194]: [2026-06-15 09:22:36.588037842] server f.000000.r.222222 fencing rid 1111111111111111 at IP x.x.x.x for quorum_block_leader
Jun 15 09:23:09 kernel: scoutfs f.000000.r.222222 error: 30000 ms recovery timeout expired for client rid 1111111111111111, fencing
Jun 15 09:23:09 kernel: sysfs: cannot create duplicate filename '/fs/scoutfs/f.000000.r.222222/fence/1111111111111111'
Jun 15 09:23:09 kernel: scoutfs f.000000.r.222222 error: fence returned err -17, shutting down server
The reclaim path keys off the rid, not the reason, so one pending fence
covers both needs. Skip submitting a duplicate fence for an rid that
already has one pending, and treat a -EEXIST from the sysfs create as
success to also cover a race or a stale rid-named dir.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Add trace events to aid debugging recovery stalls:
- scoutfs_recov_prepare(sb, rid, which):
Server registered that a client needs to recover something.
- scoutfs_recov_finish(sb, rid, which, remaining):
Client finished one recovery item, and lists remaining count.
- scoutfs_recov_timeout_fire(sb, nr_pending):
The 30s recovery window expired, with count of clients that
failed to recover within the timeout.
- scoutfs_recov_fence_rid(sb, rid):
One event per client about to be fenced.
- scoutfs_lock_server_greeting(sb, rid, recov_pending):
Shows which rid we expect to recover locks.
Signed-off-by: Auke Kok <auke.kok@versity.com>
The trace events added in the subsequent change take recinf->lock
while in timer_callback() to iterate recinf->pending, but it does
so in softirq context. To do this safely, it has to use the _bh()
spin_lock variant. Convert the other lock takers, too.
```
================================
WARNING: inconsistent lock state
5.14.0-427.35.1.el9_4.x86_64+debug #1 Tainted: G OE ------- ---
--------------------------------
inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W} usage.
swapper/2/0 [HC0[0]:SC1[1]:HE1:SE0] takes:
ffff88813cdd9c20 (&recinf->lock){+.?.}-{2:2}, at: timer_callback+0x26/0x380 [scoutfs]
{SOFTIRQ-ON-W} state was registered at:
__lock_acquire+0x7d0/0x1900
lock_acquire+0x1da/0x640
_raw_spin_lock+0x34/0x80
scoutfs_recov_finish+0x80/0x830 [scoutfs]
server_greeting+0x244/0xe60 [scoutfs]
scoutfs_net_proc_worker+0x28a/0xce0 [scoutfs]
recv_one_message+0x7e3/0xd10 [scoutfs]
scoutfs_net_recv_worker+0x441/0xe00 [scoutfs]
process_one_work+0x8e5/0x1530
worker_thread+0x598/0xf70
kthread+0x2a4/0x350
ret_from_fork+0x29/0x50
irq event stamp: 549813370
hardirqs last enabled at (549813370): [<ffffffffabe25cb4>] _raw_spin_unlock_irq+0x24/0x50
hardirqs last disabled at (549813369): [<ffffffffabe2594e>] _raw_spin_lock_irq+0x5e/0x90
softirqs last enabled at (549813356): [<ffffffffabe28c91>] __do_softirq+0x621/0x9c2
softirqs last disabled at (549813363): [<ffffffffa9a44665>] __irq_exit_rcu+0x185/0x230
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&recinf->lock);
<Interrupt>
lock(&recinf->lock);
*** DEADLOCK ***
```
Signed-off-by: Auke Kok <auke.kok@versity.com>
Scoutfs print segfaults walking a log_merge btree that has more than one
level. print_btree_block() prints parent (level > 0) items via
print_block_ref(), which invokes the item callback with a NULL value to
print the key portion before printing the child ref:
func(key, 0, 0, NULL, 0, arg);
Leaf-item printers handle this (e.g. print_srch_root_item guards with
"if (val)"), but print_log_merge_item immediately casts val and reads a
field, dereferencing NULL. Filesystems whose log_merge is a single leaf
block (height 1) never hit the parent path; one with height > 1 crashes
on the first parent item.
Observed on a system whose log_merge is height 2. The faulting address
is a small fixed offset (NULL + 8), not a wild pointer, and the crash
is in print_log_merge_item called from print_btree_block:
[Mon Jun 15 10:46:17 2026] scoutfs[22043]: segfault at 8 ip 0000000000408ef0 sp 00007fffc5edd8b0 error 4 in scoutfs[402000+e000] likely on CPU 29 (core 9, socket 1)
#0 0x0000000000408ef0 in print_log_merge_item ()
#1 0x000000000040958d in print_btree_block.constprop.0.isra ()
#2 0x000000000040a471 in print_cmd ()
#3 0x0000000000404264 in cmd_execute ()
#4 0x00000000004025c9 in main ()
The print stops at the first level-1 item (a 16-byte block_ref, matching
sizeof(struct scoutfs_block_ref)); the "[0]" line is emitted before the
callback at print_btree_block, and neither a decoded item line nor the
"ref blkno" line that print_block_ref prints after the callback appears,
so the crash is inside the callback with val == NULL:
log_merge btree blkno 11477029
crc 858a064c fsid 0000000000000000 seq 13289998818052716408 blkno 11477029
total_item_bytes 7848 mid_free_len 57640
level 1 nr_items 109 item_root.node 3600
[0] off 104 par 128 l 0 r 0 h 1 vo 65504 vl 16
Segmentation fault
Return early when val is NULL, and bounds-check val_len before each cast
so a short or malformed item is reported instead of read past its end.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This depends on nfs-utils being installed on the host. Without it
it will skip, and count as a failure. It starts nfs-server and
does a bare exportfs.
- Tests basic read/write/stage/release/data wait.
- Tests setfacl/getfacl.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Scoutfs has supported posix ACLs through the xattr handler table,
which allowed NFS to fetch them through this sideband, which worked
for older kernels.
With recent changes we've pulled in .get_acl because the mainline
kernel is changing how ACL ops are called. But we still left .set_acl
unreachable. This meant that on el9.7 nfs clients could now reach
.get_acl, but still not set them.
With this change, we're finally exposing .set_acl consistently
across all el releases and allowing nfs clients to both get and set
posix ACLs.
Signed-off-by: Auke Kok <auke.kok@versity.com>
El9.8 backported the upstream v6.15.* rename of from_timer to
timer_container_of. Switch the two callers in fence.c and recov.c
to the new style and add a simple kcompat define for older kernels.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Ramps up data preallocation based on the number of online
blocks. This results in a simple 2<<n block allocation pattern
until n=11 (2048) - the default value of data_prealloc_blocks.
Signed-off-by: Auke Kok <auke.kok@versity.com>
When scoutfs_setattr truncates a file with offline extents, it unlocks
the inode lock before calling scoutfs_data_wait to wait for the data
to be staged. If data_wait returns any error, the code jumps to 'goto
out' which calls scoutfs_unlock again, thus double-unlocking the lock.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Skews a totl twice, restore it, and intersperse setfattr/unlink to
exercise both injected and naturally-produced deltas.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Run concurrent quota add/del on one mount against rapid file
creation and deletion on both mounts to exercise the race fixed
in the previous commit.
Signed-off-by: Auke Kok <auke.kok@versity.com>
A quota check holds the quota cluster lock for READ and marks the
cached ruleset EBUSY while loading rules. A quota mod on the same
mount holds the lock for WRITE (compatible with the local READ)
and calls scoutfs_quota_invalidate(), tripping
BUG_ON(rs == ERR_PTR(-EBUSY)).
Make invalidate skip EBUSY so the reader's claim is preserved, and
have scoutfs_quota_mod_rule wait for the reader to finish before
calling invalidate. Without the wait, the in-flight reader would
publish its stale ruleset after invalidate runs, leaving the cache
stale until the next invalidation.
Signed-off-by: Auke Kok <auke.kok@versity.com>
When searching for the next least merge range we need to sweep all the
stored items because they're interleaved with respect to key sorting
because we've clobbered the zone.
To search all of them we need to start from 0, not from the caller's
start key after setting the zone. If the caller happens to provide a
start key with a small zone but large other fields (totl keys with
sufficiently large identifiers) we can miss ranges.
Signed-off-by: Zach Brown <zab@zabbo.net>
I'm seeing consistent CPU soft lockups in block_free_work on
my bare metal system that aren't reached by VM instances. The
reason is that the bare metal machine has a ton more memory
available causing the block free work queue to grow much
larger in size, and then it has so much work that it can take 30+
seconds before it goes through it all.
This is all with a debug kernel. A non debug kernel will likely
zoom through the outstanding work here at a much faster rate.
Signed-off-by: Auke Kok <auke.kok@versity.com>
block_submit_bio will return -ENOLINK if called during a forced
shutdown, the bio is never submitted, and thus no completion callback
will fire to set BLOCK_BIT_ERROR. Any other task waiting for this
specific bp will end up waiting forever.
To fix, fall through to the existing block_end_io call on the
error path instead of returning directly. That means moving
the forcing_unmount check past the setup calls so block_end_io's
bookkeeping stays balanced. block_end_io then sets BLOCK_BIT_ERROR
and wakes up waiters just as it would on a failed async completion.
Signed-off-by: Auke Kok <auke.kok@versity.com>
scoutfs_quota_mod_rule calls scoutfs_item_create/delete which use
the transaction allocator but it never held it. Without the hold,
a concurrent transaction commit can call scoutfs_alloc_init to
reinitialize the allocator while dirty_alloc_blocks is in the middle
of setting up the freed list block. This overwrites alloc->freed with
the server's fresh (empty) state, causing a blkno mismatch BUG_ON
in list_block_add.
Reproduced by stressing concurrent quota add/del operations across
mounts. Crashdump analysis confirms dirty_list_block COW'd a freed
block (fr_old=9842, new blkno=9852) but by the time list_block_add
ran, freed.ref.blkno was 0 with first_nr=0 and total_nr=0: the freed
list head had been zeroed by a concurrent alloc_init.
Fix by adding scoutfs_hold_trans/scoutfs_release_trans around the
item modification in scoutfs_quota_mod_rule, preventing transaction
commit from racing with the allocator use.
Rename the 'unlock' label to 'release' since 'out' now directly
does the unlock. The unlock safely handles a NULL lock.
Signed-off-by: Auke Kok <auke.kok@versity.com>
A malformed message encountered here increases the counter, but doesn't
tear down the connection because of the nested for loops. The comments
indicate that that is the expected behavior - a misbehaving client
should not be tolerated.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Clang's scan-build found this leak when we get an invalidation
for a lock we no longer have. Free ireq to fix.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Clang flow analysis flags resp_data in process_response as possibly
uninitialized when find_request returns NULL.
kmod/src/net.c:533:6: error: variable 'resp_data' is used uninitialized
whenever 'if' condition is false [-Werror,-Wsometimes-uninitialized]
In practice the read is harmless because resp_func stays NULL in that
path and call_resp_func only dereferences resp_data when resp_func is
non-NULL. Initialize at declaration.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Without overly broad filtering empty lines from dmesg, filter
them so dmesg.new doesn't trigger a test failure. I don't want
to overly process dmesg, so do this as late as possible.
The xfs lockdep patterns can forget a leading/trailing empty line,
causing a failure despite the explicit removal of the lockdep
false positive.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This already caught xfs_nondir_ilock_class, but recent CI runs
have been hitting xfs_dir_ilock_class, too.
Signed-off-by: Auke Kok <auke.kok@versity.com>
The gcc version in el7 can't determine that scoutfs_block_check_stale
won't return ret = 0 when the input ret value is < 0, and
errors because we might call alloc_wpage with an uninitialized
read_seq. Initialize it to 0 to avoid it.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Readers currently accumulate all finalized log tree deltas into
a single bucket for deciding whether they are already in fs_root
or not, but, finalized trees that aren't inputs to a current merge
will have higher seqs, and thus we may be double applying deltas
already merged into fs_root.
To distinguish, scoutfs_totl_merge_contribute() needs to know the
merge status item seq. We change wkic's get_roots() from using the
SCOUTFS_NET_CMD_GET_ROOTS RPC to reading the superblock directly.
This is needed because totl merge resolution has to use the same data
as the btree roots it is operating on, thus we can't grab it from a
SCOUTFS_NET_CMD_GET_ROOTS packet - it likely is different.
Signed-off-by: Auke Kok <auke.kok@versity.com>
These mislabeled members and enums were clearly not describing
the actual data being handled and obfuscating the intent of
avoiding mixing merge input items with non-merge input items.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Before deltas were added this code path was correct, but with
deltas we can't just retry this without clearing &root, since
it would potentially double count.
The condition where this could happen is when there are deltas in
several finalized log trees, and we've made progress towards reading
some of them, and then encounter a stale btree block. The retry
would not clear the collected trees, apply the same delta as was
already applied before the retry, and thus double count.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Two different clients can write delta's for totl indexes at the same
time, recording their changes. When merged, a reader should apply both
in order, and only once. To do so, the seq determines whether the delta
has been applied already.
The code fails to update the seq while walking the trees for deltas to
apply. Subsequently, when processing subsequent trees, it could
re-process deltas already applied. In case of a large negative delta
(e.g. removal of large amounts of files), the totl value could become
negative, resulting in quota lockout.
The fix is simple: advance the seq when reading partial delta merges
to avoid double counting.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Add a trigger that forces btree_merge() to return -ERANGE after
modifying a leaf's worth of items, causing many small partial merges
per merge cycle. This is used by tests to reliably reproduce races
that depend on partial merges splicing items into fs_root while
finalized logs still exist.
The trigger check lives inside btree_merge() where it can observe
actual item modification progress, rather than overriding the
caller's dirty byte limit argument which applies to the whole
writer context.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Auke Kok
Zach Brown