The tests were using high ephemeral port numbers for the mount server's
listening port. This caused occasional failure if the client's
ephemeral ports happened to collide with the ports used by the tests.
This ports all the port number configuration in one place and has a
quick check to make sure it doesn't wander into the current ephemeral
range. Then it updates all the tests to use the chosen ports.
Signed-off-by: Zach Brown <zab@versity.com>
The `-R` option will shuffle the order in which tests are executed.
The testing order shouldn't affect the outcome of any of the tests, but
in practice many of these tests will execute code slightly different
based on the history of the filesystem, resources allocated, memory
usage etc. of tests that were executed before. Shuffling the order of
tests therefore introduces small semi-random variations in the
enviroment.
The xfstests test is the only one that can't be shuffled yet into the
mix, so it is kept at the end. This is because it leaves the filesystems
unmounted. At a later point we may want to address this.
Signed-off-by: Auke Kok <auke.kok@versity.com>
If run without `-m` (explicit mkfs) in subsequent testing, old test
data files may break several tests. Most failures are -EEXIST, but
there are some more subtle ones.
This change erases any existing test dir as needed just before we
run the tests, and avoids the issue entirely.
I considered doing a `mv dir dir.$$ && rm -rf dir.$$ &` alternative
solution but that likely will interfere disproportionally with
tests that do disconnects and other thing that can be impacted by an
unlink storm.
This has an obvious performance aspect - tests will be a little
slower to start on subsequent runs. In CI, this will effectively be
a no-op though.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Stored as `results/scoutfs.tap`, this file contains TAP format 14
generated test results.
Embedded in the output are some metadata so that these files can be
aggregated and stored in an unique and deduplicating way, but using a
generated UUID at the start of testing. The file itself also catches git
ID, date, and kernel version, as well as the (possibly altered) test
sequence used.
Any test that has diff or dmesg output will be considered failed, and a
copy of the relevant data is included as comments.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Previously, any t_skip would cause the final test result to be a failure
because up until now no test should have been skipped.
However, with format-version-forward-back not being compatible with el9,
we are going to rely on el7/8 testing for that test soleley, and
therefore we have to allow skipping of this test on el9 and newer OS
versions.
We add `t_skip_permitted` to signal this from the test case to the
run-tests.sh script. A new exit code is passed, and all accounting is
updated to reflect that a test was skipped, but this was permitted. We
modify format-version-forward-back to use this new exit path.
Signed-off-by: Auke Kok <auke.kok@versity.com>
This is done by xfstests and it's so much easier to follow what is going
on from logs or e.g. serial console that I thought I should do this for
scoutfs tests as well. It makes it so much easier to discern which test
may have been cause for issues when running a bunch of tests and you're
looking back at logs later.
Signed-off-by: Auke Kok <auke.kok@versity.com>
Add a run-tests -V option that passes through the -V option to mkfs so
that runs can specify the format version that the primary volume will
have. This doesn't affect the scratch file system versions.
Signed-off-by: Zach Brown <zab@versity.com>
We don't directly mount the underlying devices for each mount because
the kernel notices multiple mounts and doesn't setup a new super block
for each.
Previously the script used loopback devices to create the local shared
block construct 'cause it was easy. This introduced corruption of
blocks that saw concurrent read and write IOs. The buffered kernel file
IO paths that loopback eventually degrades into by default (via splice)
could have buffered readers copying out of pages without the page lock
while writers modified the page. This manifest as occasional crc
failure of blocks that we knowingly issue concurrent reads and writes to
from multiple mounts (the quorum and super blocks).
This changes the script to use device-mapper linear passthrough devices.
Their IOs don't hit a caching layer and don't provide an opportunity to
corrupt blocks.
Signed-off-by: Zach Brown <zab@versity.com>
In older versions of coreutils, quoted strings are occasionally
output using utf-8 open/close single quotes.
New versions of coreutils will exclusively use the ASCII single quote
character "'" when the output is not a TTY - as is the case with
all test scripts.
We can avoid most of these problems by always setting LC_ALL=C in
testing, however.
Signed-off-by: Auke Kok <auke.kok@versity.com>
The fence script we use for our single node multi-mount tests only knows
how to fence by using forced unmount to destroy a mount. As of now, the
tests only generate failing nodes that need to be fenced by using forced
unmount as well. This results in the awkward situation where the
testing fence script doesn't have anything to do because the mount is
already gone.
When the test fence script has nothing to do we might not notice if it
isn't run. This adds explicit verification to the fencing tests that
the script was really run. It adds per-invocation logging to the fence
script and the test makes sure that it was run.
While we're at it, we take the opportunity to tidy up some of the
scripting around this. We use a sysfs file with the data device
major:minor numbers so that the fencing script can find and unmount
mounts without having to ask them for their rid. They may not be
operational.
Signed-off-by: Zach Brown <zab@versity.com>
The local-force-unmount fenced fencing script only works when all the
mounts are on the local host and it uses force unmount. It is only
used in our specific local testing scripts. Packaging it as an example
lead people to believe that it could be used to cobble together a
multi-host testing network, however temporary.
Move it from being in utils and packged to being private to our tests so
that it doesn't present an attractive nuisance.
Signed-off-by: Zach Brown <zab@versity.com>
The test harness might as well use all cpus when building. It's
reasonably safe to assume both that the test systems are otherwise idle
and that the build is likely to succeed.
Signed-off-by: Zach Brown <zab@versity.com>
We've grown some test names that are prefixes of others
(createmany-parallel, createmany-parallel-mounts). When we're searching
for lines with the test name we have to search for the exact test name,
by terminating the name with a space, instead of searching for a line
that starts with the test name.
This fixes strange output and saved passed stats for the names that
share a prefix.
Signed-off-by: Zach Brown <zab@versity.com>
We can lose interesting state if the mounts are unmounted as tests fail,
only unmount if all the tests pass.
Signed-off-by: Zach Brown <zab@versity.com>
Weirdly, run-tests was treating trace_printk not as an option to enable
trace_printk() traces but as an option to print trace events to the
console with printk? That's not a thing.
Make -P really enable trace_printk tracing and collect it as it would
enabled trace events. It needs to be treated seperately from the -t
options that enable trace events.
While we're at it treat the -P trace dumping option as a stand-alone
option that works without -t arguments.
Signed-off-by: Zach Brown <zab@versity.com>
run-tests.sh has a -t argument which takes a whitespace seperated string
of globs of events to enable. This was hard to use and made it very
easy to accidentally expand the globs at the wrong place in the script.
This makes each -t argument specify a single word glob which is stored
in an array so the glob isn't expanded until it's applied to the trace
event path. We also add an error for -t globs that didn't match any
events and add a message with the count of -t arguments and enabled
events.
Signed-off-by: Zach Brown <zab@versity.com>
We were checking for the wrong magic value.
We now need to use -f when running mkfs in run-tests for things to work.
Signed-off-by: Andy Grover <agrover@versity.com>
When we had three repos the run-tests harness helped by checking
branches in kmod and utils repos to build and test. Now that we have
one repo we can just use the sibling kmod/ and utils/ dirs in the repo.
Signed-off-by: Zach Brown <zab@versity.com>
The tests were checking that the literal string was zero, which it never
was. Once we check the value of the variable then we notice that the
sense of some tests went from -n || to -n &&, so switch those to -z.
Signed-off-by: Zach Brown <zab@versity.com>
For xfstests, we need to be able to specify both for scratch device as
well.
using -e and -f for now, but we should really be switching to long options.
Signed-off-by: Andy Grover <agrover@versity.com>
[zab@versity.com: minor arg message fixes]
Add -z option to run-tests.sh to specify metadata device.
Do a bunch of things twice.
Fix up setup-error-teardown test.
Signed-off-by: Andy Grover <agrover@versity.com>
[zab@versity.com: minor arg message fixes, golden output]
It can be handy to skip checking out specific branches from the
required repos, so -s option will skip doing so for kmod/utils/xfstests.
Also fix utils die messages to reference -U/u instead of -K/k.
Signed-off-by: Andy Grover <agrover@versity.com>
The dmesg check was creating false positives when unexpected messages
from before the test run were forced out of the ring. The evicted
messages were showing up as removals in the diff.
We only want to see new messages that were created during the test run.
So we format the diff to only output added lines.
Signed-off-by: Zach Brown <zab@versity.com>
We add directories of our built binaries for tests to find. Let's
prepend them to PATH so that we find them before any installed
binaries in the system.
Signed-off-by: Zach Brown <zab@versity.com>
Add a -y argument so we can specify additional args to ./xfstests, and
clean up our xfstest a bit while we're in there.
Signed-off-by: Zach Brown <zab@versity.com>
When running a test we only create the test dir through one mount, but
we were off-by-one when deciding that we were iterating through the
first mount.
Signed-off-by: Zach Brown <zab@versity.com>
We can't use cmd() to create the results dir because it tries to
redirect output to the results dir, which fails, so mkdir isn't run and
we don't create the results dir.
Signed-off-by: Zach Brown <zab@versity.com>
We check out the specified git branch with "origin/" prepended, but we
weren't verifying that same full branch so the verification failed
because it couldn't distinguish differentiate amongst possible named
branches.
Signed-off-by: Zach Brown <zab@versity.com>
The first commit of the scoutfs-tests suite which uses multiple mounts
on one host to test multi-node scoutfs.
Signed-off-by: Zach Brown <zab@versity.com>
