Previously we'd avoided full extents in file data mapping items because
we were deleting items from forest btrees directly. That created
deletion items for every version of file extents as they were modified.
Now we have the item cache which can remove deleted items from memory
when deletion items aren't necessary.
By layering file data extents on an extent layer, we can also transition
allocators to use extents and fix a lot of problems in the radix block
allocator.
Most of this change is churn from changing allocator function and struct
names.
File data extents no longer have to manage loading and storing from and
to packed extent items at a fixed granularity. All those loops are torn
out and data operations now call the extent layer with their callbacks
instead of calling its packed item extent functions. This now means
that fallocate and especially restoring offline extents can use larger
extents. Small file block allocation now comes from a cached extent
which reduces item calls for small file data streaming writes.
The big change in the server is to use more root structures to manage
recursive modification instead of relying on the allocator to notice and
do the right thing. The radix allocator tried to notice when it was
actively operating on a root that it was also using to allocate and free
metadata blocks. This resulted in a lot of bugs. Instead we now double
buffer the server's avail and freed roots so that the server fills and
drains the stable roots from the previous transaction. We also double
buffer the core fs metadata avail root so that we can increase the time
to reuse freed metadata blocks.
The server now only moves free extents into client allocators when they
fall below a low threshold. This reduces the shared modification of the
client's allocator roots which requires cold block reads on both the
client and server.
Signed-off-by: Zach Brown <zab@versity.com>
Add an allocator which uses btree items to store extents. Both the
client and server will use this for btree blocks, the client will use it
for srch blocks and data extents, and the server will move extents
between the core fs allocator btree roots and the clients' roots.
Signed-off-by: Zach Brown <zab@versity.com>
Add infrastructure for working with extents. Callers provide callbacks
which operate on their extent storage while this code performs the
fiddly splitting and merging of extents. This layer doesn't have any
persitent structures itself, it only operates on native structs in
memory.
Signed-off-by: Zach Brown <zab@versity.com>
It can be handy to skip checking out specific branches from the
required repos, so -s option will skip doing so for kmod/utils/xfstests.
Also fix utils die messages to reference -U/u instead of -K/k.
Signed-off-by: Andy Grover <agrover@versity.com>
bulk_create_paths was inspired by createmany when it was outputting
status lines every 10000 files. That's far too often if we're creating
files very quickly. And it only tried to output a line after entire
directories, so output could stall for very large directories.
Behave more inline with vmstat, iostat, etc, and output a line at a
regular time interval.
Signed-off-by: Zach Brown <zab@versity.com>
Add options to bulk_create_paths for creating xattrs as we create files.
We can create normal xattrs, or .srch. tagged xattrs where all, some, or
none of the files share the same xattr name.
Signed-off-by: Zach Brown <zab@versity.com>
The test that exercises re-reading stale cached blocks was still
trying to use both tiny btree blocks and segments, both of which have
been removed.
Signed-off-by: Zach Brown <zab@versity.com>
The calculation of the last valid data blkno was off by one. It was
calculating the total number of small blocks that fit in the device
size.
Signed-off-by: Zach Brown <zab@versity.com>
The recent cleanup of the radix allocator included removing tracking of
the first set bits or references in blocks.
Signed-off-by: Zach Brown <zab@versity.com>
Track the kernel changes to use the scoutfs_key struct as the btree key
instead of a big-endian binary blob.
Signed-off-by: Zach Brown <zab@versity.com>
The kernel has long sinced moved away from symbolic printing of key
cones and types, and it just removed the MAX values from the format
header. Let's follow suit and get rid of the zone and type strings.
Signed-off-by: Zach Brown <zab@versity.com>
We had manually implemented a few of the functions to add values to
specific endian types. Make a macro to generate the function and
generate them for all the endian types we use.
Signed-off-by: Zach Brown <zab@versity.com>
The percpu_counter library merges the per-cpu counters with a shared
count when the per-cpu counter gets larger than a certain value. The
default is very small, so we often end up taking a shared lock to update
the count. Use a larger batch so that we take the lock less often.
Signed-off-by: Zach Brown <zab@versity.com>
Now that the item cache is bearing the load of high frequency item
calls, we can remove all the item granular work that the forest was
trying to do. The item cache amortizes the cost of the forest so its
remaining methods can go straight to the btrees and don't need
complicated state to reduce the overhead of item calls.
Signed-off-by: Zach Brown <zab@versity.com>
Use the new item cache for all the item work in the fs instead of
calling into the forest of btrees. Most of this is mechanical
conversion from the _forest calls to the _item calls. The item cache
no longer supports the kvec argument for describing values so all the
callers pass in the value pointer and length directly.
The item cache doesn't support saving items as they're deleted and later
restoring them from an error unwinding path. There were only two users
of this. Directory entries can easily guarantee that deletion won't
fail by dirtying the items first in the item cache. Xattr updates were
a little trickier. They can combine dirtying, creating, updating, and
deleting to atomically switch between items that describe different
versions of a multi-item value. This also fixed a bug in the srch
xattrs where replacing an xattr would create a new id for the xattr and
leave existing srch items referencing a now deleted id. Replacing now
reuses the old id.
And finally we add back in the locking and transaction item cache
integration.
Signed-off-by: Zach Brown <zab@versity.com>
Add an item cache between fs callers and the forest of btrees. Calling
out to the btrees for every item operation was far too expensive. This
gives us a flexible in-memory structure for working with items that
isn't bound by the constrants of persistent block IO. We can rarely
stream large groups of items to and from the btrees and then use
efficient kernel memory structures for more frequent item operations.
This adds the infrastructure, nothing is calling it yet.
Signed-off-by: Zach Brown <zab@versity.com>
Add forest calls that the item cache will use. It needs to read all the
items in the leaf blocks of forest btree which could contain the key,
write dirty items to the log btree, and dirty bits in the bloom block as
items are dirtied.
Signed-off-by: Zach Brown <zab@versity.com>
Add btree calls to call a callback for all items in a leaf, and to
insert a list of items into their leaf blocks. These will be used by
the item cache to populate the cache and to write dirty items into dirty
btree blocks.
Signed-off-by: Zach Brown <zab@versity.com>
The current btree walk recorded the start and end of child subtrees as
it walked, and it could give the caller the next key to iterate towards
after the block it returned. Future methods want to get at the key
bounds of child subtrees, so we add a key range struct that all walk
callers provide and fill it with all the interesting keys calculated
during the walk.
Signed-off-by: Zach Brown <zab@versity.com>
Btree traversal doesn't split a block if it has room for the caller's
item. Extract this test into a function so that an upcoming btree call
can test that each of multiple insertions into a leaf will fit.
Signed-off-by: Zach Brown <zab@versity.com>
Remove the last remnants of the indexed xattrs which used fs items.
This makes the significant change of renumbering the key zones so I
wanted it in its own commit.
Signed-off-by: Zach Brown <zab@versity.com>
In a merge where the input and source trees are the same, the input
block can be an initial pre-cow version of the dirty source block.
Dirtying blocks in the change will clear allocations in the dirty source
block but they will remain in the pre-cow input block. The merge can
then set these blocks in the dst, even though they were also used by
allocation, because they're still set in the pre-cow input block.
This fix is clumsy, but minimal and specific to this problem. A more
thorough fix is being worked on which introduces more staging more
allocator trees and should stop calls that are modifying the current
active avail or free trees.
Signed-off-by: Zach Brown <zab@versity.com>
Lock invalidation has to make sure that changes are visible to future
readers. It was syncing if the current transaction is dirty. This was
never optimal, but it wasn't catastrophic when concurrent invalidation
work could all block on one sync in progress.
With the move to a single invalidation worker serially invalidating
locks it became unacceptable. Invalidation happening in the presence of
writers would constantly sync the current transaction while very old
unused write locks were invalidated. Their changes had long since been
committed in previous transactions.
We add a lock field to remember the transaction sequence which could
have been dirtied under the lock. If that transaction has already been
comitted by the time we invalidate the lock it doesn't have to sync.
Signed-off-by: Zach Brown <zab@versity.com>
The client lock network message processing callbacks were built to
simply perform the processing work for the message in the networking
work context that it was called in. This particularly makes sense for
invalidation because it has to interact with other components that
require blocking contexts (syncing commits, invalidating inodes,
truncating pages, etc).
The problem is that these messages are per-lock. With the right
workloads we can use all the capacity for executing work just in lock
invalidation work. There is no more work execution available for other
network processing. Critically, the blocked invalidation work is
waiting for the commit thread to get its network responses before
invalidation can make forward progress. I was easily reproducing
deadlocks by leaving behind a lot of locks and then triggering a flood
of invalidation requests on behalf of shrinking due to memory pressure.
The fix is to put locks on lists and have a small fixed number of work
contexts process all the locks pending for each message type. The
network callbacks don't block, they just put the lock on the list and
queue the work that will walk the lists. Invalidation now blocks one
work context, not the number of incoming requests.
There were some wait conditions in work that used to use the lock workq.
Other paths that change those conditions now have to know to queue the
work specifically, not just wake tasks which included blocked work
executors.
The other subtle impact of the change is that we can no longer rely on
networking to shutdown message processing work that was happening in its
callbacks. We have to specifically stop our work queues in _shutdown.
Signed-off-by: Zach Brown <zab@versity.com>
While checking for lost server commit holds, I noticed that the
advance_seq request path had obviously incorrect unwinding after getting
an error. Fix it up so that it always unlocks and applies its commit.
Signed-off-by: Zach Brown <zab@versity.com>
Add the committed_seq to statfs_more which gives the greatest seq which
has been committed. This lets callers disocover that a seq for a change
they made has been committed.
Signed-off-by: Zach Brown <zab@versity.com>
We had a debugging WARN_ON that warns when a client has an error
commiting their transaction. Let's add a bit more detail and promote it
to a proper error. These should not happen.
Signed-off-by: Zach Brown <zab@versity.com>
The forest code is responsible for constructing a consistent fs image
out of the items spread across all the btrees written by mounts in the
system.
Usually readers walk a btree looking for log trees that they should
read. As a mount modifies items in its dirty log tree, readers need to
be sure to check that in-memory dirty log tree even though it isn't
present in the btree that records persistent log trees.
The code did this by setting a flag to indicate that readers using a
lock should check the dirty log tree. But the flag usage wasn't
properly locked and left a race where a reader and writer could race,
leaving future readers to not know that they should check the dirty log
tree. When we rarely hit that race we'd see item errors that made no
sense, like not being able to find an inode item to update after having
just created it in the current transaction.
To fix this, we clean up the tree tracking in the forest code.
We get rid of the static forest_root structs in the lock_private that
were used to track the two special-case roots that aren't found in log
tree items: the in-memory dirty log root and the final fs root. All
roots are now dynamically allocated. We use a flag in the root to
identify it as the dirty log root, and identify the fs root by its
rid/nr. This results in a bunch of caller churn as we remove lpriv from
root identifying functions.
We get rid of the idea of the writer adding a static root to the list as
well as marking the log as needing to read the root. Instead we make
all root management happen as we refresh the list. The forest maintains
a commit sequence and writers set state in the lock to indicate that the
lock has dirty items in the log during this transaction. Iteration then
compares the state set by the commit, writer, and the last refresh to
determine if a new refresh needs to happen.
Properly tracking the presence of dirty items lets us recognize when the
lock no longer has dirty items in the log and we can stop locking and
reading the dirty log and fall back to reading the committed stable
version. The previous code didn't do that, it would lock and read the
dirty root forever.
While we're in here, we fix the locking around setting bloom bits and
have it track the version of the log tree that was set so that we don't
have to clear set bits as the log version is rotated out by the server.
There was also a subtle bug where we could hit to stale errors for the
same root and return -EIO because we triggering refresh returned stale.
We rework the retrying logic to use a separate error code to force
refreshing so that we can't accidentally trigger eio by conflating
reading stale blocks and forcing refreshing.
And finally, we no longer record that we need the dirty log tree in a
root if we have a lock that could never read. It's a minor optimization
that doesn't change functional behaviour.
Signed-off-by: Zach Brown <zab@versity.com>
Using strictly coherent btree items to map the hash of xattr names to
inode numbers proved the value of the functionality, but it was too
expensive. We now have the more efficient srch infrastructure to use.
We change from the .indx. to the .srch. tag, and change the ioctl from
find_xattr to search_xattrs. The idea is to communicate that these are
accelerated searches, not precise index lookups and are relatively
expensive.
Rather than maintaining btree items, xattr setting and deleting emits
srch entries which either tracks the xattr or combines with the previous
tracker and removes the entry. These are done under the lock that
protects the main xattr item, we can remove the separate locking of the
previous index items.
The semantics of the search ioctl needs to change a bit. Because
searches are so expensive we now return a flag to indicate that the
search completed. While we're there, we also allow a last_ino parameter
so that searches can be divided up and run in parallel.
Signed-off-by: Zach Brown <zab@versity.com>
This introduces the srch mechanism that we'll use to accelerate finding
files based on the presence of a given named xattr. This is an
optimized version of the initial prototype that was using locked btree
items for .indx. xattrs.
This is built around specific compressed data structures, having the
operation cost match the reality of orders of magnitude more writers
than readers, and adopting a relaxed locking model. Combine all of this
and maintaining the xattrs no longer tanks creation rates while
maintaining excellent search latencies, given that searches are defined
as rare and relatively expensive.
The core data type is the srch entry which maps a hashed name to an
inode number. Mounts can append entries to the end of unsorted log
files during their transaction. The server tracks these files and
rotates them into a list of files as they get large enough. Mounts have
compaction work that regularly asks the server for a set of files to
read and combine into a single sorted output file. The server only
initiates compactions when it sees a number of files of roughly the same
size. Searches then walk all the commited srch files, both log files
and sorted compacted files, looking for entries that associate an xattr
name with an inode number.
Signed-off-by: Zach Brown <zab@versity.com>
The get_fs_roots rpc and server interfaces were built around individual
roots. Rebuild it around passing around a struct so that we can add
roots without impacting all the current users.
Signed-off-by: Zach Brown <zab@versity.com>
Now that we have larger blocks we can have a larger max item. This was
increased to make room for the srch compaction items which store a good
number of srch files in their value.
Signed-off-by: Zach Brown <zab@versity.com>
The conversion of the super block metadata block counters to units of
large metadata blocks forgot to scale back to the small block size when
filling out the block count fields in the statfs rpc. This resulted in
the free and total metadata use being off by the factor of large to
small block size (default of ~16x at the moment).
Signed-off-by: Zach Brown <zab@versity.com>
Zach BrownandZach Brown
Andy GroverandZach Brown