This script executes the basic parallel_restore test binary which
incorporates the parallel restore library. The test binary creates
a few files with xattrs. After restoring, we mount the filesystem
and do some basic checks to see that the restore was complete.
Added just under and just over ENOSPC cases to make sure that we are
returning the final 5% of this disk that we reserver for log trees.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Signed-off-by: Hunter Shaffer <hunter.shaffer@versity.com>
Adds a function to to insert quota rules as filesystem items.
This will then have an outward facing function that takes a
writer and a mirror of the _squota_rule struct in quota.c
and is called _parallel_restore_quota_rule. Adds testing
to make sure we are restoring a test quota.
Signed-off-by: Hunter Shaffer <hunter.shaffer@versity.com>
Adds a check in the inode creation path that checks whether
the retention feature is present. If it is then we set the
inode's flag value otherwise it stays as 0.
Signed-off-by: Hunter Shaffer <hunter.shaffer@versity.com>
Project IDs add a new field 'proj' to the inode struct.
In this patch we simply check if the feature is present
before we compile and if it is we allocate the field
within the parallel_restore inode and make sure this
value set in the restored inode. If it is not present
we ignore it.
Signed-off-by: Hunter Shaffer <hunter.shaffer@versity.com>
The filesystem we are restoring into needs to be empty
and never mounted. Here we check the all of the quorum blocks
timestamps to see whether the device we are restoring into has
been mounted before. Adds a test in the test script that attempts
to restore a previously mounted device.
Signed-off-by: Hunter Shaffer <hunter.shaffer@versity.com>
Signed-off-by: Auke Kok <auke.kok@versity.com>
When we import the superblock we first check whether
the filesystem is at least format version 2. We then
verify that the device given is a meta_dev. The
test script now also initialize the new filesystem
to format_V2. Adds coverage in the test script for the
new checks.
Signed-off-by: Hunter Shaffer <hunter.shaffer@versity.com>
This tool compies a source tree (whether it's scoutfs or not)
into an offline scoutfs meta device. It has only those 2 parameters
and does a single-process walk of the tree to restore all items
while preservice as much of the metadata as possible.
Signed-off-by: Auke Kok <auke.kok@versity.com>
After this change, all files have a single offline extent:
```
$ sudo src/filefrag-gc57857a5 -b4096 -v /mnt/scratch/top-0/file-1094
Filesystem type is: 554f4353
File size of /mnt/scratch/top-0/file-1094 is 4096 (1 block of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 0: 0.. 0: 1: last,unknown_loc,eof
/mnt/scratch/top-0/file-1094: 1 extent found
```
Signed-off-by: Auke Kok <auke.kok@versity.com>
Adds a signal handler for SIGCHLD and sets up a signal handler with
SA_SIGINFO. This way we can inspect the exit code emitted by the
child and abort processing when a child process exits with an error.
Without this handler, any child process that exits with e.g. ENOSPC
will keep the parent hanging indefinitely.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This allows callers to pass in seq values for generated inodes. The
tester code initializes them now before calling, instead of being
hard set in the library.
Signed-off-by: Auke Kok <auke.kok@versity.com>
While online extents (non-zero size) worked just fine with this
code, the offline extent code inserted a btree item without the
appropriate key, which results in duplicate (null) keys being
inserted, hence the "duplicate" error.
All that is needed to fix is to put the created key in the btree
item to be inserted.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This block never called insert_fs_item() creating dangling keys
that never got inserted. Additionally, the _sk_second member is
le64 and we have to use the proper intrinsic to increment it.
Signed-off-by: Auke Kok <auke.kok@versity.com>
The list alloc blocks have an array of blknos that are offset by a start
field in the block header. The print code wasn't using that and was
always referencing the beginning of the array, which could miss blocks.
Signed-off-by: Zach Brown <zab@versity.com>
We check superblock magic, crc, flags. data device superblock is
checked but a little less thorough. We check whether the device is
still mounted, since that would make checking invalid to begin with.
Quorum blocks are validated to have sane contents.
We add a global problem counter so we can trivially measure and
report whether any problem was found at all, instead of iterating
over all the problems and checking each individual count.
We pick the standard exit code values from `fsck` and mirror their
intentional behavior. This results in `fsck.scoutfs` can now be
trivially created by making it a wrapper around `scoutfs check`.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Signed-off-by: Hunter Shaffer <hunter.shaffer@versity.com>
Generally as we call block_get() we should validate that if the block
has a hdr, at a minimum the crc is correct and the magic value is
the expected value passed, and the fsid matches the superblock. This
function implements just that. Returns -EINVAL, up to the caller to
report a problem() and handle the outcome. For now the code just hard
fails, which incedentally makes it fail the clobber-repair.sh tests
I wrote.
Signed-off-by: Auke Kok <auke.kok@versity.com>
As I was committing the initial check command I had only partially
completed a rename of the function that checks the metadata allocators.
Signed-off-by: Zach Brown <zab@versity.com>
Previously, any t_skip would cause the final test result to be a failure
because up until now no test should have been skipped.
However, with format-version-forward-back not being compatible with el9,
we are going to rely on el7/8 testing for that test soleley, and
therefore we have to allow skipping of this test on el9 and newer OS
versions.
We add `t_skip_permitted` to signal this from the test case to the
run-tests.sh script. A new exit code is passed, and all accounting is
updated to reflect that a test was skipped, but this was permitted. We
modify format-version-forward-back to use this new exit path.
Signed-off-by: Auke Kok <auke.kok@versity.com>
I'm seeing more and more of these as audit is enabled in el8 and el9
images I am using for testing, and during ENOSPC tests this has a chance
of triggering process accounting suspension, and subsequent resume.
Signed-off-by: Auke Kok <auke.kok@versity.com>
In v1.18-10-g5507ee5, we changed the test code away from loopback
to device-mapper, which simplified our DUT setup code.
However, this results in the occasional `device changed size` messages
now being emitted by the `dm` driver instead of the `loop` kernel
module. We have to additionally ignore these kernel messages from now as
well.
Signed-off-by: Auke Kok <auke.kok@versity.com>
In v5.17-rc4-53-g3a3bae50af5d, we can no longer omit having this
method unhooked as the mm caller blindly calls it now. In-kernel
filesystems all were fixed in this change.
aops->invalidatepage was the old aops method that would free pages
with private attached data. This method is replaced with the
new invalidate_folio method. If this method is NULL, the memory
will become orphaned. (v5.17-rc4-29-gf50015a596fa)
Signed-off-by: Auke Kok <auke.kok@versity.com>
I've pushed a tag/release to scoutfs-xfstests-dev instead of a full
blown branch. This seems simpler and cleaner than using branches,
because we're going to end up rebasing these things a lot. However, we
can't --track tags, so, if the branch name passed to -x is actually a
tag instead of a branch, we have to omit the --track option here.
Signed-off-by: Auke Kok <auke.kok@versity.com>
CI testing needs to know which xfstests branch to use on all OSs.
We can't just use the el9 xfstests branch on el9 only, because we
need to run the same el9 xfstests on el8 and el7 as well, otherwise
testing will just fail.
So, we put a marker file in our git repo that tells us that we're
not going to use the default `scoutfs` branch from scoutfs-xfstests-dev
but our own special tag or branch. The CI job then should pass the
proper -x {branch} flag to the run-tests.sh script.
Signed-off-by: Auke Kok <auke.kok@versity.com>
The new version of xfstests adds a _lot_ more tests to our mix. Many
of the new ones will auto enable or auto skip as needed.
There are tests we can't or won't support that will be in future
xfstests. Disable them now so we can avoid dealing with them later.
Quite a few fall into "we don't support these types of mounting yet",
mostly bind-mount or dm-mapper things. We disable all the swapfile
tests flatout.
A few tests fail on el7 but not el8/9 but we don't have a way to run
them without failing yet, so disable them as well.
Update golden with the proper new array of tests. This all requires
the `auke/scoutfs-el9` branch in `versity/scoutfs-xfstests-dev`.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Using t_skip, we just skip this test on el9.
If we ever want to add a formatversion 2->3 test, perhaps we should
just add a separate test script, instead of going over a static array.
But let's not worry about this too much right now.
Signed-off-by: Auke Kok <auke.kok@versity.com>
It turns out that on el9, `bash -c` prints out `bash: line 1: cd..`
instead of `line 0:` on el7 or el8. So discard all the stderr from
these `cd` lines entirely and just rely on the expected echo
output to stdout.
Signed-off-by: Auke Kok <auke.kok@versity.com>
There's filefrag already, and that works, but, it's output is very
inconsistent between various OS release versions, and it has already
meant that we'd needed to adjust tests to account for these little
but insignificant changes. A lot more work than useful. It's even
more changed in el9.
This adds `scoutfs get-fiemap FILE` and prints out block extent
info with flags that we care about as an abbreviated letter: U for
Unwritten, L for Last, and O for Unknown (as in, "offline").
The -P/--physical and -L/--logical options turn off logical or physical
offset display, in case you only want to see the offsets in either
units. You can pass -b/--byte to display offsets and lengths in
byte values. The block size will then be obtained from fstat() of
the queried file (4096 for scoutfs).
I've removed all uses of filefrag from our scoutfs tests. Xfstests
still calls it but their internal diff takes care of that issue.
Where needed and appropriate, the tests are adjusted so that the output
of `scoutfs get-fiemap` is as close as it can to what it used to be,
so that reading the test results allows the quick view of what might
have been going wrong.
There are some output strings I have not bothered to update because
there's no real value to updating every output string to match,
and we just adjust the golden file accordingly.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This isn't a simple case where we can use u64_region_wraps because
length is s32.
Let's actually test an overflow case instead of a case that doesn't
overflow, though. We still should properly add an overflow test here as
well.
Signed-off-by: Auke Kok <auke.kok@versity.com>
We use check_add_overflow(a, b, d) here to validate that (off, len)
pairs do not exceed the max value type. The kernel conveniently has
several macros to sort out the problems with signed or unsigned types.
However, we're not interested in purely seeing whether (a + b)
overflows, because we're using this for (off, len) overflow checks,
where the bytes we read are from 0 to len -1. We must therefore call
this check with (b) being "len - 1".
I've made sure that we don't accidentally fail when (len == 0)
in all cases by making sure we've already checked this condition
before, and moving code around as needed to ensure that (len > 0)
in all cases where we check.
The macro check_add_overflow requires a (d) argument in which
temporarily the result of the addition is stored and then checked to see
if an overflow occurred. We put a `tmp` variable on the stack of the
correct type as needed to make the checks function.
simple-release-extents test mistakenly relied on this buggy wrap code,
so it needs fixing. The move-blocks test also got it wrong.
Signed-off-by: Auke Kok <auke.kok@versity.com>
We consistently enter scoutfs_data_wait_check when len == 0 from
scoutfs_aio_write() which directly passes the i_size_read() value,
and for cases where we `echo >> $FILE` this is always reached.
This can cause the wrapping check to fail since `0 + (0 - 1) < 0` which
triggers the WARN_ON_ONCE wrap check that needs updating to allow
certain operations on huge files.
More importantly we can just omit all these checks if `len == 0` anyway,
since they should always succeed and never should require taking all the
locks.
Signed-off-by: Auke Kok <auke.kok@versity.com>
