CI can run several test.py sessions on different machines (builders) for
one build and, and to be not overwritten, .db file with metrics need to
have some unique name: add host_id as we already do for .xml report in
run_pytest()
Also add host_id columns to metric tables in case we will somehow
aggregate .db files.
Implement repetition of files using pytest_collect_file hook: run
file collection as many times as needed to cover all --mode/--repeat
combinations. Also move disabled test logic to this hook.
Store build mode and run_id in pytest item stashes.
Simplify support of C++ tests: remove redundant facade abstraction and put
all code into 3 files: base.py, boost.py, and unit.py
Add support for `run_first` option in test_config.yaml
Cmake emits its build.ninja into build/, while configure.py emits
build.ninja into ./. test.py uses this difference to choose the directory
structure to test.
The problem is that vscode will randomly call cmake to understand the
directory structure, so we end up with both build.ninja set up.
Invert the logic to look for ./build.ninja to determine the mode (instead
of build/build.ninja which can exist even if the user uses traditional
configuration).
It can still happen that a stray ./build.ninja exists (for example due
to switching branches), but that is rarer than having vscode auto-create
it.
Closesscylladb/scylladb#24269
Add path constants to `test` module and use them in different test suites
instead of own dups of the same code:
- TOP_SRC_DIR : ScyllaDB's source code root directory
- TEST_DIR : the directory with test.py tests and libs
- BUILD_DIR : directory with ScyllaDB's build artefacts
Add TestSuite.log_dir attribute as a ScyllaDB's build mode subdir of a path
provided using `--tmpdir` CLI argument. Don't use `tmpdir` name because it
mixed up with pytest's built-in fixture and `--tmpdir` option itself.
Change default value for `--tmdir` from `./testlog` to `TOP_SRC_DIR/testlog`
Refactor `ResourceGather*` classes to use path from a `test` object instead of
providing it separately.
Move modes constants to `test` module and remove duplications.
Move `prepare_dirs()` and `start_3rd_party_services()` from `pylib.util` to
`pylib.suite.base` to avoid circular imports (with little refactoring to
use `pathlib.Path` instead of `str` as paths.)
Also, in some places refactor to use f-strings for formatting.
Move starting LDAP to the method where the rest of the services are
started. This will unify the way of starting the 3rd party services.
Fix LDAP tests flakiness due not possible to connect to LDAP server
Add catching stdout and stderr of toxiproxy-cli in case of errors
Allow starting clusters of Scylla servers. Chain up the next
server start to the end of the previous one, and set the next
server's seed to the previous server.
As a workaround for a race between token dissemination through
gossip and streaming, change schema version to force a gossip
round and make sure all tokens end up at the joining node in time.
Make sure scylla start is not race prone.
auth::standard_role_manager creates "cassandra" role in an async loop
auth::do_after_system_ready(), which retries role creation with an
exponential back-off. In other words, even after CQL port is up, Scylla
may still be initializing.
This race condition could lead to spurious errors during cluster
bootstrap or during a test under CI.
When the role is ready, queries begin to work, so rely on this "side
effect".
To start or stop servers, use a new class, ScyllaCluster,
which encapsulates multiple servers united into a cluster.
In it, validate that a test case cleans up after itself.
Additionally, swallow startup errors and throw them when
the test is actually used.
