Make sure that data is regularly synced. We switch to a delayed work
struct that is always queued with the sync deadline. If we need an
immediate sync we mod it to now.
Signed-off-by: Zach Brown <zab@versity.com>
This was mostly just a demonstration for how to add messages. We're
about to add a message that we always send on mount so this becomes
completely redundant.
Signed-off-by: Zach Brown <zab@versity.com>
When inserting a newly allocated segment we might find an existing
cached stale segment. We replace it in the cache so that its user can
keep using its stale contents while we work on the new segment.
Replacing doesn't clear the rb_node, though, so we trip over a warning
when we finally free the segment and it looks like it's still present in
the rb tree.
Clear the node after we replace it so that freeing sees a clear node and
doesn't issue a warning.
Signed-off-by: Zach Brown <zab@versity.com>
Add support for the inode index items which are replacing the seq walks
from the old btree structures. We create the index items for the root
inode, can print out the items, and add a commmand to walk the indices.
Signed-off-by: Zach Brown <zab@versity.com>
Recent kernel headers have leaked __bitwise into userspace. Rename our
use of __bitwise in userspace sparse builds to avoid the collision.
Signed-off-by: Zach Brown <zab@versity.com>
It's a bit confusing to always see both the old and current super block.
Let's only print the first one. We could add an argument to print all
of them.
Signed-off-by: Zach Brown <zab@versity.com>
Add items for indexing inodes by their fields. When we update the inode
item we also delete the old index items and create the new items. We
rename and refactor the old inode since ioctl to now walk the inode
index items.
Signed-off-by: Zach Brown <zab@versity.com>
Now that we have basic file extents we can add a flag to extents to
track offline extents. We have to initialize and test the flags as we
work with extents. Truncation can be told to leave removed extents
around with no block mapping and the offline bit set. Only staging with
the correct data version can write to the offline regions. Demand
staging isn't implemented yet. Reads from offline extents are treated
like sparse regions.
Truncation is a straight forward iteration over the portions of existing
extents which overlap with the truncated blocks.
Writing to offline extents has to first remove the existing offline
extent before then adding the new allocated extents. The 'changes'
mechanism relied on being able to search the current items to find the
changes that should be made before making any changes. This doesn't
work for finding merge candidates for the new allocated insertion
because the old offline extent change won't have been applied yet. We
replace the change mechanism with straight forward item modification and
unwinding.
The generic block fiemap can't communicate offline extents and iterates
over blocks instead of extents. We add our fiemap that iterates
over extents and sets the 'UNKNOWN' flag on offline extents.
Signed-off-by: Zach Brown <zab@versity.com>
The offset comparison in release that was meant to catch wrapping was
inverted and accidentally prevented releasing a single block.
Signed-off-by: Zach Brown <zab@versity.com>
xattrs can have 0 lenth values so fix the item iterator to emit a single
item in the case where the size is 0.
Signed-off-by: Zach Brown <zab@versity.com>
kill_sb tries to sync before calling kill_block_super. It shouldn't do
this on mount errors that wouldn't have initialized the higher level
systems needed for syncing.
Signed-off-by: Zach Brown <zab@versity.com>
The ring node rb walker was returning an exact match for the search key
instead of the last node that was traversed. This stopped callers from
then iterating from the traversed node to find the next or previous
node.
Signed-off-by: Zach Brown <zab@versity.com>
Our first attempt at storing file data put them in items. This was easy
to implement but won't be acceptable in the long term. The cost of the
power of LSM indexing is compaction overhead. That's acceptable for
fine grained metadata but is totally unacceptable for bulk file data.
This switches to storing file data in seperate block allocations which
are referenced by extent items.
The bulk of the change is the mechanics of working with extents. We
have high level callers which add or remove logical extents and then
underlying mechanisms that insert, merge, or split the items that
the extents are stored in.
We have three types of extent items. The primary type maps logical file
regions to physical block extents. The next two store free extents
per-node so that clients don't create lock and LSM contention as they
try and allocate extents.
To fill those per-node free extents we add messages that communcate free
extents in the form of lists of segment allocations from the server.
We don't do any fancy multi-block allocation yet. We only allocate
blocks in get_blocks as writes find unmapped blocks. We do use some
per-task cursors to cache block allocation positions so that these
single block allocations are very likely to merge into larger extents as
tasks stream wites.
This is just the first chunk of the extent work that's coming. A later
patch adds offline flags and fixes up the change nonsense that seemed
like a good idea here.
The final moving part is that we initiate writeback on all newly
allocated extents before we commit the metadata that references the new
blocks. We do this with our own dirty inode tracking because the high
level vfs methods are unusably slow in some upstream kernels (they walk
all inodes, not just dirty inodes.)
Signed-off-by: Zach Brown <zab@versity.com>
The dirent name formatting code accidentally copied the calculation for
the length of the name from the xattrs, which are null terminated. The
durents are not, their length is just the value length minus the dirent
header.
Signed-off-by: Zach Brown <zab@versity.com>
The compaction code is still directly referencing the super block
and calling sync methods as though it was still standalone. This is
mostly OK because only the server runs it. But it isn't quite right
because the sync methods no longer make the rings persistent as they
write the item transaction. The server is in control of that now.
Eventually we'll have compaction messages being sent between the mount
clients and the server. Let's take a step in that direction by having
the compaction work call net methods to get its compaction parameters
and finish the compaction. Eventually these would be marshalled through
request/process/reply code.
But in this first step we know that the compaction code is running on
the server so we can forgo all the messaging and just call in to and out
of compaction. The net calls just holds the ring consistency locks in
the server and call into the manifest to do the work, commiting the
changes when its done.
This is more careful about segno alloction and freeing. Compaction
doesn't call the allocator directly. It gets allocaitons from the
messages and returns them if it doesn't use them. We actually now
free segnos as they're removed from the manifest.
With the server controlling compaction and can tear all the fiddly level
count watching code out of the manifest. Item transactions can't care
about the level counts and the server always tries compaction after the
manifest is updated intead of having the manifest watch the level counts
and call compaction.
Now that the server owns the rings they should not be torn down as the
super is torn down, net does that now. And we need to be more careful
to be sure that writes from dirtying and compaction are stable before
killing the super.
With all this in place moving to shared compaction involves adding the
messages and negotiating concurrent compactions in the manifest.
Signed-off-by: Zach Brown <zab@versity.com>
Add mkfs and print support for the simpler rings that the segment bitmap
allocator and manifest are now using. Some other recent format header
updates come along for the ride.
Signed-off-by: Zach Brown <zab@versity.com>
The segment item struct used to have fiddly packed offsets and lengths.
Now it's just normal fields so we can work with them directly and get
rid of the native item indirection.
Signed-off-by: Zach Brown <zab@versity.com>
We were using a bitmap to record segments during manifest printing and
then walking that bitmap to print segments. It's a little silly to have
a second data structure record the referenced segments when we could
just walk the manifest again to print the segments.
So refactor node printing into a treap walker that calls a function for
each node. Then we can have functions that print the node data
structurs for each treap and then one that prints the segments that are
referenced by manifest nodes.
Signed-off-by: Zach Brown <zab@versity.com>
We had changed the manifest keys to fully cover the space around the
segments in the hopes that it'd let item reading easily find negative
cached regions around items.
But that makes compaction think that segments intersect with items when
they really don't. We'd much rather avoid unnecessary compaction by
having the manifest entries precisely reflect the keys in the segment.
Item reading can do more work at run time to find the bounds of the key
space that are around the edges of the segments it works with.
Signed-off-by: Zach Brown <zab@versity.com>
Make sure that the manifest entries for a given level fully
cover the possible key space. This helps item reading describe
cached key ranges that extend around items.
Signed-off-by: Zach Brown <zab@versity.com>
Update mkfs and print to describe the ring blocks with a starting index
and number of blocks instead of a head and tail index.
Signed-off-by: Zach Brown <zab@versity.com>
Make a new file system by writing a root inode in a segment and storing
a manifest entry in the ring that references the segment.
Signed-off-by: Zach Brown <zab@versity.com>
Inode info struct initialization spread out over three places:
- once for the memory of a slab obect
- when reading an existing inode from items
- when initializing a newly allocated inode
Over time field initializtion got out of sync with these rules. This
makes it more clear which fields get initialized where. In the inode
info struct we group fields by where there initialized. We order the
fields by size and location in the inode struct.
Then we make sure that all the initialization sites have everything
covered. Doing everything in consistent struct order makes it easier
to audit that we haven't missed anything.
What lead to this was realizing that we missed initializing the seqcount
when reading existing inodes. It should have been initialized in the
slab object constructor. The 'staging' boolean has the same problem.
Signed-off-by: Zach Brown <zab@versity.com>
Reviewed-by: Mark Fasheh <mfasheh@versity.com>
For consistency and to keep upstream users (scout-utils, etc) from
needing to include different type headers, we'll change the type to
match the rest of the header.
Signed-off-by: Nic Henke <nic.henke@versity.com>
Add support for more of the known xattr namespaces. This helps
generic/062 in xfstests pass.
Signed-off-by: Zach Brown <zab@versity.com>
Reviewed-by: Mark Fasheh <mfasheh@versity.com>
xfstests generic/062 was failing because it was getting an unexpected
error code when trying to work with xattrs on special files. Adding our
ops gives it the errnos it expects.
Signed-off-by: Zach Brown <zab@versity.com>
Reviewed-by: Mark Fasheh <mfasheh@versity.com>
Zach Brown
Nic HenkeandZach Brown