126 lines
4.1 KiB
Markdown
126 lines
4.1 KiB
Markdown
# Testing
|
|
|
|
## Introduction
|
|
|
|
`test.py` is a simple regression testing harness shipped along with
|
|
scylla.git, which runs C++ unit and CQL tests.
|
|
|
|
This is a manual for `test.py`.
|
|
|
|
|
|
## Installation
|
|
|
|
To run `test.py`, Python 3.7 or higher is required.
|
|
`./install-dependencies.sh` should install all the required Python
|
|
modules. If `install-dependencies.sh` does not support your distribution,
|
|
please manually install all Python modules it lists with `pip`.
|
|
|
|
|
|
## Usage
|
|
|
|
In order to invoke `test.py`, you need to build first. `./test.py` will
|
|
run all existing tests in all configured build modes:
|
|
|
|
$ ./test.py
|
|
|
|
If you want to specify a specific build mode:
|
|
|
|
$ ./test.py --mode=dev
|
|
|
|
If you want to run only a specific test:
|
|
|
|
$ ./test.py suitename/testname
|
|
|
|
Build artefacts, such as test output and harness output is stored
|
|
in `./testlog`. Scylla data files are stored in `/tmp`.
|
|
|
|
## How it works
|
|
|
|
On start, `test.py` invokes `ninja` to find out configured build modes. Then
|
|
it searches all subdirectories of `./test/` for `suite.yaml` files: each
|
|
directory containing `suite.yaml` is a test suite, in which `test.py` then looks
|
|
for tests. All files ending with `_test.cc` or `_test.cql` are considered
|
|
tests.
|
|
|
|
A suite must contain tests of the same type, as configured in `suite.yaml`.
|
|
The list of found tests is matched with the optional command line test name
|
|
filter. A match is registered if filter substring exists anywhere in test
|
|
full name. For example:
|
|
|
|
$ ./test.py cql
|
|
|
|
runs `cql/lwt_test`, `cql/lwt_batch_test`, as well as
|
|
`boost/cql_query_test`.
|
|
|
|
The `./testlog` directory is created if it doesn't exist, otherwise it is
|
|
cleared from the previous run artefacts.
|
|
|
|
Matched tests are run concurrently, with concurrency factor set to the
|
|
number of available CPU cores. `test.py` continues until all tests are run
|
|
even if any one of them fails.
|
|
|
|
## CQL tests
|
|
|
|
To run CQL tests, `test.py` uses an auxiliary program, `test/tools/cql_repl.cc`.
|
|
This program reads CQL input from stdin, evaluates it using Scylla CQL
|
|
database front-end, and prints output in JSON format to stdout. A default
|
|
keyspace is created automatically.
|
|
|
|
`test.py` invokes `cql_repl` providing the test file for its standard
|
|
input and redirecting its output to a temporary file in `testlog` directory.
|
|
|
|
After `cql_repl` finishes, `test.py` compares the output stored in the
|
|
temporary file with a pre-recorded output stored in
|
|
`test/suitename/testname_test.result`.
|
|
|
|
The test is considered failed if executing any CQL statement produced an
|
|
error (e.g. because the server crashed during execution) or server output
|
|
does not match one recorded in `testname.result`, or there is no
|
|
`testname.result`. The latter is possible when it's the first invocation of
|
|
the test ever.
|
|
|
|
In the event of output mismatch file `test/suitename/testname_test.reject`
|
|
is created, and first lines of the diff between the two files are output.
|
|
To update `.result` file with new output, simply overwrite it with the
|
|
reject file:
|
|
|
|
mv test/suitename/testname.re*
|
|
|
|
*Note* Since the result file is effectively part of the test, developers
|
|
must thorougly examine the diff and understand the reasons for every
|
|
change before overwriting `.result` files.
|
|
|
|
## Unit tests
|
|
|
|
The same unit test can be run in different seastar configurations, i.e. with
|
|
different command line arguments. The custom arguments can be set in
|
|
`custom_args` key of the `suite.yaml` file.
|
|
|
|
|
|
## Debugging
|
|
|
|
If a test fails, its log can be found in `testlog/${mode}/testname.log`.
|
|
By default, all unit tests are built stripped. To build non-stripped tests,
|
|
`./configure` with `--tests-debuginfo list-of-tests`.
|
|
`test.py` adds some command line arguments to unit tests. The exact way in
|
|
which the test is invoked is recorded in `testlog/test.py.log`.
|
|
|
|
To debug CQL tests, one can invoke `gdb cql_repl`, which will start
|
|
`cql_repl` in interactive mode, and then supply faulty CQL statements
|
|
on the terminal.
|
|
|
|
|
|
## Automation
|
|
|
|
If any of the tests fails, `test.py` returns a non-zero exit status.
|
|
JUNIT and XUNIT execution status XML files can be found in
|
|
`testlog/${mode}/xml/` directory.
|
|
|
|
|
|
## See also
|
|
|
|
For command line help and available options, please see also:
|
|
|
|
$ ./test.py --help
|
|
|