3fda13e20e
We have been seeing rare failures of the cql-pytest (translated from
Cassandra's unit tests) for testing TTL in secondary indexes:
cassandra_tests/validation/entities/secondary_index_test.py::testIndexOnRegularColumnInsertExpiringColumn
The problem is that the test writes an item with 1 second TTL, and then
sleeps *exactly* 1.0 seconds, and expects to see the item disappear
by that time. Which doesn't always happen:
The problem with that assumption stems from Scylla's TTL clock ("gc_clock")
being based on Seastar's lowres clock. lowres_clock only has a 10ms
"granularity": The time Scylla sees when deciding whether an item expires
may be up to 10ms in the past - the arbitrary point when the lowres timer
happened to last run. In rare overload cases, the inaccuracy may be even
grater than 10ms (if the timer got delayed by other things running).
So when Scylla is asked to expire an item in 1 second - we cannot be
sure it will be expired in exactly 1 second or less - the expiration
can be also around 10ms later.
So in this patch we change the test to sleep with more than enough
margin - 1.1 seconds (i.e., 100ms more than 1 second). By that time
we're sure the item must have expired.
Before this patch, I saw the test failing once every few hundred runs,
after this patch I ran if 2,000 times without a single failure.
Fixes #9008
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210712100655.953969-1-nyh@scylladb.com>
Single-node functional tests for Scylla's CQL features.
These tests use the Python CQL library and the pytest frameworks. By using an actual CQL library for the tests, they can be run against any implementation of CQL - both Scylla and Cassandra. Most tests - except in rare cases - should pass on both, to ensure that Scylla is compatible with Cassandra in most features.
To run all tests against an already-running local installation of Scylla
or Cassandra on localhost, just run pytest. The "--host" and "--port"
can be used to give a different location for the running Scylla or Cassanra.
The "--ssl" option can be used to use an encrypted (TLSv1.2) connection.
More conveniently, we have two scripts - "run" and "run-cassandra" - which do all the work necessary to start Scylla or Cassandra (respectively), and run the tests on them. The Scylla or Cassandra process is run in a temporary directory which is automatically deleted when the test ends.
"run" automatically picks the most recently compiled version of Scylla in
build/*/scylla - but this choice of Scylla executable can be overridden with
the SCYLLA environment variable. "run-cassandra" defaults to running the
command cassandra from the user's path, but this can be overriden by setting
the CASSANDRA environment variable to the path of the cassandra script,
e.g., export CASSANDRA=$HOME/apache-cassandra-3.11.10/bin/cassandra.
A few of the tests also require the nodetool when running on Cassandra -
this tool is again expected to be in the user's path, or be overridden with
the NODETOOL environment variable.
Additional options can be passed to "pytest" or to "run" / "run-cassandra" to control which tests to run:
- To run all tests in a single file, do
pytest test_table.py. - To run a single specific test, do
pytest test_table.py::test_create_table_unsupported_names.
Additional useful pytest options, especially useful for debugging tests:
- -v: show the names of each individual test running instead of just dots.
- -s: show the full output of running tests (by default, pytest captures the test's output and only displays it if a test fails)