055340ae39
In commit 7eda6b1e90, we increased the
request_timeout parameter used by cql-pytest tests from the default of
10 seconds to 120 seconds. 10 seconds was usually more than enough for
finishing any Scylla request, but it turned out that in some extreme
cases of a debug build running on an extremely over-committed machine,
the default timeout was not enough.
Recently, in issue #11289 we saw additional cases of timeouts which
the request_timeout setting did *not* solve. It turns out that the Python
CQL driver has two additional timeout settings - connect_timeout and
control_connection_timeout, which default to 5 seconds and 2 seconds
respectively. I believe that most of the timeouts in issue #11289
come from the control_connection_timeout setting - by changing it
to a tiny number (e.g., 0.0001) I got the same error messages as those
reported in #11289. The default of that timeout - 2 seconds - is
certainly low enough to be reached on an extremely over-committed
machine.
So this patch significantly increases both connect_timeout and
control_connection_timeout to 60 seconds. We don't care that this timeout
is ridiculously large - under normal operations it will never be reached.
There is no code which loops for this amount of time, for example.
Refs #11289 (perhaps even Fixes, we'll need to see that the test errors
go away).
NOTE: This patch only changes test/cql-pytest/util.py, which is only
used by the cql-pytest test suite. We have multiple other test suites which
copied this code, and those test suites might need fixing separately.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes #11295
Single-node functional tests for Scylla's CQL features.
These tests use the Python CQL library and the pytest frameworks. By using an actual CQL library for the tests, they can be run against any implementation of CQL - both Scylla and Cassandra. Most tests - except in rare cases - should pass on both, to ensure that Scylla is compatible with Cassandra in most features.
To run all tests against an already-running local installation of Scylla
or Cassandra on localhost, just run pytest. The "--host" and "--port"
can be used to give a different location for the running Scylla or Cassanra.
The "--ssl" option can be used to use an encrypted (TLSv1.2) connection.
More conveniently, we have two scripts - "run" and "run-cassandra" - which do all the work necessary to start Scylla or Cassandra (respectively), and run the tests on them. The Scylla or Cassandra process is run in a temporary directory which is automatically deleted when the test ends.
"run" automatically picks the most recently compiled version of Scylla in
build/*/scylla - but this choice of Scylla executable can be overridden with
the SCYLLA environment variable. "run-cassandra" defaults to running the
command cassandra from the user's path, but this can be overriden by setting
the CASSANDRA environment variable to the path of the cassandra script,
e.g., export CASSANDRA=$HOME/apache-cassandra-3.11.10/bin/cassandra.
A few of the tests also require the nodetool when running on Cassandra -
this tool is again expected to be in the user's path, or be overridden with
the NODETOOL environment variable. Nodetool is not needed to test
Scylla.
Additional options can be passed to "pytest" or to "run" / "run-cassandra" to control which tests to run:
- To run all tests in a single file, do
pytest test_table.py. - To run a single specific test, do
pytest test_table.py::test_create_table_unsupported_names. - To run the same test or tests 100 times, add the
--count=100option. This is faster than runningrun100 times, because Scylla is only run once, and also counts for you how many of the runs failed. Forpytestto support the--countoption, you need to install a pytest extension:pip install pytest-repeat
Additional useful pytest options, especially useful for debugging tests:
- -v: show the names of each individual test running instead of just dots.
- -s: show the full output of running tests (by default, pytest captures the test's output and only displays it if a test fails)