Commit Graph

60 Commits

Author SHA1 Message Date
Zach Brown
ec702b9bb3 Update the data_version ioctl to return the u64
We updated the code to use the new iteration of the data_version ioctl
but we forgot to update the ioctl definition so it didn't actually work.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-17 15:45:56 -08:00
Zach Brown
c3f122a5f1 Fix mkfs buddy initialization
mkfs was starting setting free blk bits from 0 instead of from
the blkno offset of the first free block.  This resulted in
the highest order above a used blkno being marked free.  Freeing
that blkno would set its lowest order blkno.  Now that blkno can be
allocated from two orders.  That, eventually, can lead to blocks
being doubly allocated and users trampling on each other.

While auditing the code to chase this bug down I also noticed that
write_buddy_blocks() was using a min() that makes no sense at all.  Here
'blk' is inclusive, the modulo math works on its own.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-17 15:43:23 -08:00
Zach Brown
932b0776d1 Add commands for working with offline data
Add the data_version, stage, and release commands for working with
offline extents of file data.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-16 14:40:09 -08:00
Zach Brown
22140c93d1 Print extents instead of bmap items
Print the extent items now that we're not using bmap items any more.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-16 14:40:04 -08:00
Zach Brown
c2cfb0227f Print the new inode data_version field
We've added a data_version field to the inode for tracking changes to
the file's data.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-16 14:39:39 -08:00
Zach Brown
f1d8955303 Support the ino_path ioctl
We updated the inode_paths ioctl to return one path and use a counter
cursor and renamed it to ino_path.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-16 14:39:00 -08:00
Zach Brown
fb16af7b7d btree nr_items is now a le16
The btree block now has a le16 nr_items field to make room for the
number of items that larger blocks can hold.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-16 14:38:04 -08:00
Zach Brown
cd0d045c93 Add support for full radix buddy blocks
Update mkfs and print for the full radix buddy allocators.  mkfs has to
calculate the number of blocks and the height of the tree and has to
initialize the paths down the left and right side of the tree.
Print needs to dump the new radix blockx and super block fields.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-04 14:16:25 -07:00
Zach Brown
40b9f19ec4 Add bitops.c for find_next_bit_le()
The upcoming buddy changes are going to need a find_next_bit_le().

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-04 14:16:25 -07:00
Zach Brown
871db60fb2 Add U16_MAX
Add a simple U16_MAX define for upcoming buddy changes.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-04 14:16:25 -07:00
Zach Brown
a901db2ff7 Print seqs in bmap items
The bmap items now have the sequence number that wrote each mapped
block.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-04 14:16:25 -07:00
Zach Brown
b436772376 Add orphan key
Printing the raw item is enough, it doesn't have a value to decode.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-04 14:16:25 -07:00
Zach Brown
e6222223c2 Update format.h for kernel code helpers
Update format.h for some format defines that have so far only been used
by the kernel code.

Signed-off-by: Zach Brown <zab@versity.com>
2016-11-04 14:16:22 -07:00
Zach Brown
0dff7f55a6 Use openssl for pseudo random bytes
The pseudo random byte wrapper function used the intel instructions
so that it could deal with high call rates, like initializing random
node priorities for a large treap.

But this is obviously not remotely portable and has the annoying habit
of tripping up versions of valgrind that haven't yet learned about these
instructions.

We don't actually have high bandwidth callers so let's back off and just
let openssl take care of this for us.

Signed-off-by: Zach Brown <zab@versity.com>
2016-09-27 09:47:50 -07:00
Zach Brown
4ccb80a8ec Initialize all the buddy slot free order fields
Initialize the free_order field in all the slots of the buddy index
block so that the kernel will try to allocate from them and will
initialize and populate the first block.

Signed-off-by: Zach Brown <zab@versity.com>
2016-09-08 16:40:39 -07:00
Zach Brown
86ffdf24a2 Add symlink support
Print out the raw symlink items.

Signed-off-by: Zach Brown <zab@versity.com>
2016-08-29 10:25:46 -07:00
Zach Brown
a89f6c10b1 Add buddy indirect order totals
The total counts of all the set order bits in all the child buddy blocks
is needed for statfs.

Signed-off-by: Zach Brown <zab@versity.com>
2016-08-23 16:41:57 -07:00
Zach Brown
2f91a9a735 Make command listing less noisy
It's still not great, but at least it's a little clearer.

Signed-off-by: Zach Brown <zab@versity.com>
2016-08-23 12:31:03 -07:00
Zach Brown
c17a7036ed Add find xattr commands
Add commands that use the find-xattr ioctls to show the inode numbers of
inodes which probably contain xattrs matching the specified name or
value.

Signed-off-by: Zach Brown <zab@versity.com>
2016-08-23 12:21:47 -07:00
Zach Brown
43619a245d Add inode-paths via link backrefs
Add the inode-paths command which uses the ioctl to display all the
paths that lead to the given inode.  We add support for printing
the new link backref items and inode and dirent fields.

Signed-off-by: Zach Brown <zab@versity.com>
2016-08-11 16:50:57 -07:00
Zach Brown
be4a137479 Add support for printing block map items
Signed-off-by: Zach Brown <zab@versity.com>
2016-08-10 15:19:09 -07:00
Zach Brown
25e3b03d94 Add support for simpler btree block
Update mkfs and print to the new simpler btree block format.

Signed-off-by: Zach Brown <zab@versity.com>
2016-08-02 13:31:06 -07:00
Zach Brown
0af40547b5 Update to smaller block size
We're going to try using a smaller fixed block size to reduce complexity
in the file data extent code.

Signed-off-by: Zach Brown <zab@versity.com>
2016-08-02 13:30:40 -07:00
Zach Brown
6a97aa3c9a Add support for the radix buddy bitmaps
Update mkfs and print to support the buddy allocator that's indexed by
radix blocks.

Signed-off-by: Zach Brown <zab@versity.com>
2016-08-02 13:29:51 -07:00
Zach Brown
4b86256904 Ignore sparse warning for builtin fpclassify
Signed-off-by: Zach Brown <zab@versity.com>
2016-07-27 13:56:01 -07:00
Zach Brown
99167f6d66 Expand little endian bitops functions
We had the start of functions that operated on little endian bitmaps.
This adds more operations and uses __packed to support unaligned bitmaps
on platforms where unaligned accesses are a problem.

Signed-off-by: Zach Brown <zab@versity.com>
2016-07-27 13:50:51 -07:00
Zach Brown
c48e08a378 Add -fno-strict-aliasing
We modify the same memory through pointers of different types all the
live long day.

Signed-off-by: Zach Brown <zab@versity.com>
2016-07-27 13:49:27 -07:00
Zach Brown
1cacc50de0 Remove old unused lebitmap code
Signed-off-by: Zach Brown <zab@versity.com>
2016-07-22 15:04:07 -07:00
Zach Brown
fc37ece26b Remove homebrew tracing
Happily, it turns out that there are crash extensions for extracting
trace messages from crash dumps.  That's good enough for us.

Signed-off-by: Zach Brown <zab@versity.com>
2016-07-22 13:54:10 -07:00
Zach Brown
54044508fa Add inodes-since command
The kernel now has an ioctl to give us inode numbers with their sequence
number for every inode that's been modified since a given tree update
sequence number.

Update mkfs and print to the on-disk format changes and add a trivial
inodes-since command which calls the ioctl and prints the results.

Signed-off-by: Zach Brown <zab@versity.com>
2016-07-05 17:49:13 -04:00
Zach Brown
a069bdd945 Add format header updates for xattrs
The kernel now has items and structs for xattrs.

Signed-off-by: Zach Brown <zab@versity.com>
2016-07-04 11:02:07 -07:00
Zach Brown
d774e5308b Add support for printing traces from files
We can extract the formats and records from a crash dump and
print them from the files.

Signed-off-by: Zach Brown <zab@versity.com>
2016-05-28 12:41:30 -07:00
Zach Brown
54867b0f9c Add support for printing kernel traces
Add a 'trace' command which uses the debugfs file created by the scoutfs
kernel module to read and print trace messages.

Signed-off-by: Zach Brown <zab@versity.com>
2016-05-28 11:10:08 -07:00
Zach Brown
29c1f529f1 Get rid of max dirent collision nr in inode
The slightly tweaked format that uses linear probing to mitigate dirent
name hash collisions doesn't need a record of the greatest number of
collisions in the dir inode.

Signed-off-by: Zach Brown <zab@versity.com>
2016-05-02 21:40:02 -07:00
Zach Brown
67ad29508d Update for next_ino in super block
Add support for storing the next allocated inode in the super block.

Signed-off-by: Zach Brown <zab@versity.com>
2016-05-01 09:11:52 -07:00
Zach Brown
77c673f984 Add mkfs and print support for buddy alloc
Initialize the block count fields in the super block on mkfs and print
out the buddy allocator fields and blocks.

Signed-off-by: Zach Brown <zab@versity.com>
2016-04-30 12:25:59 -07:00
Zach Brown
1235f04c4a Print parent block ref item values
Signed-off-by: Zach Brown <zab@versity.com>
2016-04-14 12:59:37 -07:00
Zach Brown
56077b61a1 Move to btree blocks
Update mkfs and printing for the btree experiment.

Signed-off-by: Zach Brown <zab@versity.com>
2016-04-12 19:33:32 -07:00
Zach Brown
c4fcf40097 Update ring manifest deletion entries
The ring now contains stores full manifest entries that are deleted
rather than just their block number.

Signed-off-by: Zach Brown <zab@versity.com>
2016-04-02 20:30:45 -04:00
Zach Brown
544fd1ba9a Add ctrstat command
Like vmstat and iostat, this prints out our counters over time.

Signed-off-by: Zach Brown <zab@versity.com>
2016-04-01 00:04:26 -04:00
Zach Brown
af2975111a Update format for smaller bloom
Update our format for the smaller bloom sizes.

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-29 13:10:45 -04:00
Zach Brown
7ea78502c8 Read both super blocks and use current
When printing try to read both super blocks and use the most recent one
instead of just using the first one.

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-29 13:07:00 -04:00
Zach Brown
10cf83ffc5 Update key type value format change
Adding file data items changed the item key values.

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-26 14:00:19 -04:00
Zach Brown
339c719e4e Print dirents in print command
Add support for printing dirent items to scoutfs print.  We're careful
to change non-printable characters to ".".

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-25 00:24:48 -04:00
Zach Brown
e1c1c50ead Update to multiple dirent hash format
Update print to show the inode fields in the newer dirent hashing
scheme.  mkfs doesn't create directory entries.

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-24 21:09:51 -07:00
Zach Brown
e0e6179156 Fix bloom filter bugs
The bloom filter had two bad bugs.

First the calculation was adding the bit width of newly hashed data to
the hash value instead of the record of the hashed bits available.

And the block offset calculation for each bit wasn't truncated to the
number of bloom blocks.  While fixing this we can clean up the code and
make it faster by recording the bits in terms of their block and bit
offset instead of their large bit value.

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-23 22:23:21 -04:00
Zach Brown
ddf5ef1017 Fix set_bit_le() type width problems
The swizzle value was defined in terms of longs but the code used u64s.
And the bare shifted value was an int so it'd get truncated.  Switch it
all to using longs.

The ratio of bugs to lines of code in that first attempt was through the
roof!

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-23 22:21:11 -04:00
Zach Brown
502783e1bc Update to segment format with skiplists and bloom
Update to the format rev which has large log segments that start with
bloom filter blocks, have items linked in a skip list, and item values
stored at offsets in the block.

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-23 15:23:54 -07:00
Zach Brown
463f5e5a07 Correctly store last random word
pseudo_random_bytes() was accidentally copying the last partial long to
the beggining of the buffer instead of the end.  The final partial long
bytes weren't being filled.

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-23 15:16:58 -07:00
Zach Brown
d0429e1c88 Add minimal bloom filter helpers
mkfs just needs to initialize bloom filter blocks with the bits for the
single root inode key.  We can get away with these skeletal functions
for now.

Signed-off-by: Zach Brown <zab@versity.com>
2016-03-23 14:01:16 -07:00