Commit Graph

54 Commits

Author SHA1 Message Date
Zach Brown
3c9eeeb2ef scoutfs-utils: add transaction seq btree
Signed-off-by: Zach Brown <zab@versity.com>
2019-04-12 10:54:20 -07:00
Zach Brown
64bdda717c scoutfs-utils: move super id to block hdr magic
Move the magic value that identifies the super block into the block
header and use it for btree blocks as well.

Signed-off-by: Zach Brown <zab@versity.com>
2019-04-12 10:54:20 -07:00
Zach Brown
ea969a5dde scoutfs-utils: update format.h for quorum
Signed-off-by: Zach Brown <zab@versity.com>
2019-04-12 10:54:20 -07:00
Zach Brown
bbfa71361f scoutfs-utils: compaction request format update
Signed-off-by: Zach Brown <zab@versity.com>
2018-08-28 15:34:33 -07:00
Zach Brown
078d2f6073 scoutfs-utils: update format for greeting node_id
Signed-off-by: Zach Brown <zab@versity.com>
2018-08-28 15:34:33 -07:00
Zach Brown
7abf5c1e2b scoutfs-utils: calculate segment crc in mkfs
Signed-off-by: Zach Brown <zab@versity.com>
2018-08-21 13:27:37 -07:00
Zach Brown
ea2ec838ec scoutfs-utils: use one super and verify its crc
Signed-off-by: Zach Brown <zab@versity.com>
2018-06-29 15:56:36 -07:00
Zach Brown
59739e0057 scoutfs-utils: remove sneaky tab in mkfs output
We had a tab in the mkfs output that'd cause it to be misaligned.

Signed-off-by: Zach Brown <zab@versity.com>
2018-06-29 14:42:08 -07:00
Zach Brown
cfc8cb8800 scoutfs-utils: support server extent allocation
Signed-off-by: Zach Brown <zab@versity.com>
2018-06-29 14:42:08 -07:00
Zach Brown
98d06c7a6b scoutfs-utils: mkfs requires 16 segments
mkfs needs to make sure that a device is large enough for a file system.
We had a tiny limit that almost certainly wouldn't have worked.
Increase the limit to a still absurdly small but arguably possible 16
segments.

Signed-off-by: Zach Brown <zab@versity.com>
2018-06-29 14:42:08 -07:00
Zach Brown
8e6c18a0fa scoutfs-utils: support small keys
Make the changes to support the new small key struct.  mkfs and print
work with simpler keys, segment items, and manifest entries.  The item
cache keys ioctl now just needs to work with arrays of keys.

Signed-off-by: Zach Brown <zab@versity.com>
2018-04-04 09:15:54 -05:00
Zach Brown
7d674fa4bf scoutfs-utils: remove size inode index items
With the removal of the size index items we no longer have to print them
or be able to walk the index.  mkfs only needs to create a meta seq
index item for the root inode.

Signed-off-by: Zach Brown <zab@versity.com>
2018-01-30 15:03:50 -08:00
Mark Fasheh
7df8b87128 scoutfs-utils: cmd_register - pass a parsing friendly argv
We were chopping off the command string when passing the argument array into
registered commands. getopt expects a program name as the first argument, so
change cmd_execute() to only chop off the scoutfs program name now. Now we
can parse command arguments in an easy and standard manner.

This necessitates a small update of each commands usage of argv/argc.

Signed-off-by: Mark Fasheh <mfasheh@versity.com>
2017-11-28 14:47:50 -08:00
Zach Brown
b3d11925c7 scoutfs-utils: add support for format_hash
Calculate the hash of format.h and ioctl.h, put it in the super during
mkfs, and print it out.

Signed-off-by: Zach Brown <zab@versity.com>
2017-10-12 13:58:11 -07:00
Zach Brown
34fc095392 scoutfs-utils: update btree ring calc
Update the calculation of the largest number of btree blocks based on
the format.h update that provides the min free space in parent blocks
instead of the free limit for the entire block.

Signed-off-by: Zach Brown <zab@versity.com>
2017-10-12 13:58:11 -07:00
Mark Fasheh
2c89ff3a07 scoutfs-utils: remove inode ctime and mtime index items
These were removed in the kernel, we no longer need
them in userspace.

Signed-off-by: Mark Fasheh <mfasheh@versity.com>
2017-08-22 15:58:48 -07:00
Zach Brown
7bbe49fde2 scoutfs-utils: sort keys by zones, then types
Our item cache protocol is tied to holding DLM locks which cover a
region of the item namespace.  We want locks to cover all the data
associated with an inode and other locks to cover the indexes.  So we
resort the items first by major (index, fs) then by inode type (inode,
dirent, etc).

Signed-off-by: Zach Brown <zab@versity.com>
2017-07-19 10:33:12 -07:00
Zach Brown
6c37e3dee0 scoutfs-utils: add btree ring storage
Manifest entries and segment allocation bitmap regions are now stored in
btree items instead of the ring log.  This lets us work with them
incrementally and share them between nodes.

Signed-off-by: Zach Brown <zab@versity.com>
2017-07-17 13:43:37 -07:00
Zach Brown
6ae8e9743f scoutfs-utils: add support for skip list segments
Signed-off-by: Zach Brown <zab@versity.com>
2017-06-27 14:07:25 -07:00
Zach Brown
228c5d8b4b scoutfs-utils: support meta and data seqs
Signed-off-by: Zach Brown <zab@versity.com>
2017-05-23 12:17:52 -07:00
Zach Brown
9fc99a8c31 scoutfs-utils: add support for inode index items
Add support for the inode index items which are replacing the seq walks
from the old btree structures.  We create the index items for the root
inode, can print out the items, and add a commmand to walk the indices.

Signed-off-by: Zach Brown <zab@versity.com>
2017-05-16 11:14:10 -07:00
Zach Brown
e09a216762 Support simpler ring entries
Add mkfs and print support for the simpler rings that the segment bitmap
allocator and manifest are now using.  Some other recent format header
updates come along for the ride.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:43 -07:00
Zach Brown
a147239022 Remove dead block, btree, and buddy code
Remove the last bits of the dead code from the old btree design.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:43 -07:00
Zach Brown
acda5a3bf1 Add support for free_segs in super
The allocator records the total number of free segments in the super
block.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:43 -07:00
Zach Brown
52291b2c75 Update format for readdir_pos
We now track each parent dir's next readdir pos and the readdir pos of
each dirent.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:43 -07:00
Zach Brown
c4f2563cc1 Update tools to new segment item layout
The segment item struct used to have fiddly packed offsets and lengths.
Now it's just normal fields so we can work with them directly and get
rid of the native item indirection.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:43 -07:00
Zach Brown
e81c256a22 Remove the bitops helpers
We don't have any use for the bitops today, we'll resurrect this in
simpler form if it's needed again.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:43 -07:00
Zach Brown
26a4266964 Set manifest keys to precise segment keys
We had changed the manifest keys to fully cover the space around the
segments in the hopes that it'd let item reading easily find negative
cached regions around items.

But that makes compaction think that segments intersect with items when
they really don't.  We'd much rather avoid unnecessary compaction by
having the manifest entries precisely reflect the keys in the segment.

Item reading can do more work at run time to find the bounds of the key
space that are around the edges of the segments it works with.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:43 -07:00
Zach Brown
c2b47d84c1 Add next_seg_seq field to super
Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:43 -07:00
Zach Brown
484b34057a Update mkfs and print for treap ring
Update mkfs and print now that the manifest and allocator are stored in
treaps in the ring.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:43 -07:00
Zach Brown
7c4bc528c6 Make sure manifests cover all keys
Make sure that the manifest entries for a given level fully
cover the possible key space.  This helps item reading describe
cached key ranges that extend around items.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:42 -07:00
Zach Brown
c3b6dd0763 Describe ring log with index,nr
Update mkfs and print to describe the ring blocks with a starting index
and number of blocks instead of a head and tail index.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:42 -07:00
Zach Brown
818e149643 Update mkfs and print for lsm writing
Adapt mkfs and print for the format changes made to support writing
segments.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:42 -07:00
Zach Brown
c96b833a36 mkfs LSM segment and ring stuctures
Make a new file system by writing a root inode in a segment and storing
a manifest entry in the ring that references the segment.

Signed-off-by: Zach Brown <zab@versity.com>
2017-04-18 14:20:02 -07:00
Zach Brown
c3f122a5f1 Fix mkfs buddy initialization
mkfs was starting setting free blk bits from 0 instead of from
the blkno offset of the first free block.  This resulted in
the highest order above a used blkno being marked free.  Freeing
that blkno would set its lowest order blkno.  Now that blkno can be
allocated from two orders.  That, eventually, can lead to blocks
being doubly allocated and users trampling on each other.

While auditing the code to chase this bug down I also noticed that
write_buddy_blocks() was using a min() that makes no sense at all.  Here
'blk' is inclusive, the modulo math works on its own.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-17 15:43:23 -08:00
Zach Brown
fb16af7b7d btree nr_items is now a le16
The btree block now has a le16 nr_items field to make room for the
number of items that larger blocks can hold.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-16 14:38:04 -08:00
Zach Brown
cd0d045c93 Add support for full radix buddy blocks
Update mkfs and print for the full radix buddy allocators.  mkfs has to
calculate the number of blocks and the height of the tree and has to
initialize the paths down the left and right side of the tree.
Print needs to dump the new radix blockx and super block fields.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-04 14:16:25 -07:00
Zach Brown
4ccb80a8ec Initialize all the buddy slot free order fields
Initialize the free_order field in all the slots of the buddy index
block so that the kernel will try to allocate from them and will
initialize and populate the first block.

Signed-off-by: Zach Brown <zab@versity.com>
2016-09-08 16:40:39 -07:00
Zach Brown
a89f6c10b1 Add buddy indirect order totals
The total counts of all the set order bits in all the child buddy blocks
is needed for statfs.

Signed-off-by: Zach Brown <zab@versity.com>
2016-08-23 16:41:57 -07:00
Zach Brown
25e3b03d94 Add support for simpler btree block
Update mkfs and print to the new simpler btree block format.

Signed-off-by: Zach Brown <zab@versity.com>
2016-08-02 13:31:06 -07:00
Zach Brown
6a97aa3c9a Add support for the radix buddy bitmaps
Update mkfs and print to support the buddy allocator that's indexed by
radix blocks.

Signed-off-by: Zach Brown <zab@versity.com>
2016-08-02 13:29:51 -07:00
Zach Brown
54044508fa Add inodes-since command
The kernel now has an ioctl to give us inode numbers with their sequence
number for every inode that's been modified since a given tree update
sequence number.

Update mkfs and print to the on-disk format changes and add a trivial
inodes-since command which calls the ioctl and prints the results.

Signed-off-by: Zach Brown <zab@versity.com>
2016-07-05 17:49:13 -04:00
Zach Brown
67ad29508d Update for next_ino in super block
Add support for storing the next allocated inode in the super block.

Signed-off-by: Zach Brown <zab@versity.com>
2016-05-01 09:11:52 -07:00
Zach Brown
77c673f984 Add mkfs and print support for buddy alloc
Initialize the block count fields in the super block on mkfs and print
out the buddy allocator fields and blocks.

Signed-off-by: Zach Brown <zab@versity.com>
2016-04-30 12:25:59 -07:00
Zach Brown
56077b61a1 Move to btree blocks
Update mkfs and printing for the btree experiment.

Signed-off-by: Zach Brown <zab@versity.com>
2016-04-12 19:33:32 -07:00
Zach Brown
c4fcf40097 Update ring manifest deletion entries
The ring now contains stores full manifest entries that are deleted
rather than just their block number.

Signed-off-by: Zach Brown <zab@versity.com>
2016-04-02 20:30:45 -04:00
Zach Brown
502783e1bc Update to segment format with skiplists and bloom
Update to the format rev which has large log segments that start with
bloom filter blocks, have items linked in a skip list, and item values
stored at offsets in the block.

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-23 15:23:54 -07:00
Zach Brown
a0a3ef9675 Mark all mkfs chunks allocated in bitmap
The initial bitmap entry written in the ring by mkfs was off by one.
Three chunks were written but the 0th chunk is also free for the supers.
It has to mark the first four chunks as allocated.

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-17 17:05:24 -07:00
Zach Brown
d8f76cb893 Minor ring manifest format updates
Update to the format changes that were made while implementing ring
replay in the kernel.
2016-02-25 22:45:06 -08:00
Zach Brown
906c0186bc Get path size with stat or ioctl
If we're making a file system in a real device then we need to get
the device size with an ioctl.
2016-02-25 22:40:48 -08:00