f2978e1873
A previous patch added test/cql-pytest/cassandra_tests - a framework for porting Cassandra's unit tests to Python - but only ported two tiny test files with just 3 tests. In this patch, we finally port a much larger test file validation/entities/collection_test.java. This file includes 50 separate tests, which cover a lot of aspects of collection support, as well as how other stuff interact with collections. As of now, 23 (!) of these 50 tests fail, and exposed six new issues in Scylla which I carefully documented: Refs #7735: CQL parser missing support for Cassandra 3.10's new "+=" syntax Refs #7740: CQL prepared statements incomplete support for "unset" values Refs #7743: Restrictions missing support for "IN" on tables with collections, added in Cassandra 4.0 Refs #7745: Length of map keys and set items are incorrectly limited to 64K in unprepared CQL Refs #7747: Handling of multiple list updates in a single request differs from recent Cassandra Refs #7751: Allow selecting map values and set elements, like in Cassandra 4.0 These issues vary in severity - some are simply new Cassandra 4.0 features that Scylla never implemented, but one (#7740) is an old Cassandra 2.2 feature which it seems we did not implement correctly in some cases that involve collections. Note that there are some things that the ported tests do not include. In a handful of places there are things which the Python driver checks, before sending a request - not giving us an opportunity to check how the server handles such errors. Another notable change in this port is that the original tests repeated a lot of tests with and without a "nodetool flush". In this port I chose to stub the flush() function - it does NOT flush. I think the point of these tests is to check the correctness of the CQL features - *not* to verify that memtable flush works correctly. Doing a real memtable flush is not only slow, it also doesn't really check much (Scylla may still serve data from cache, not sstables). So I decided it is pointless. An important goal of this patch is that all 50 tests (except three skipped tests because Python has client-side checking), pass when run on Cassandra (with test/cql-pytest/run-cassandra). This is very important: It was very easy to make mistakes while porting the tests, and I did make many such mistakes; But running the against Cassandra allowed me to fix those mistakes - because the correct tests should pass on Cassandra. And now they do. Unfortunately, the new tests are significantly slower than what we've been accustomed in Alternator/CQL tests. The 50 tests create more than a hundred tables, udfs, udts, and similar slow operations - they do not reuse anything via fixtures. The total time for these 50 tests (in dev build mode) is around 18 seconds. Just one test - testMapWithLargePartition is responsibe for almost half (!) of that time - we should consider in the future whether it's worth it or can be made smaller. Signed-off-by: Nadav Har'El <nyh@scylladb.com> Message-Id: <20201215155802.2867386-1-nyh@scylladb.com>
Single-node funtional tests for Scylla's CQL features. Tests use the Python CQL library and the pytest frameworks. By using an actual CQL library for the tests, they can be run against any implementation of CQL - both Scylla and Cassandra. Most tests - except in rare cases - should pass on both, to ensure that Scylla is compatible with Cassandra in most features.
To run all tests against an already-running local installation of Scylla
or Cassandra on localhost, just run pytest. The "--host" and "--port"
can be used to give a different location for the running Scylla or Cassanra.
More conveniently, we have two scripts - "run" and "run-cassandra" - which do all the work necessary to start Scylla or Cassandra (respectively), and run the tests on them. The Scylla or Cassandra process is run in a temporary directory which is automatically deleted when the test ends.
Additional options can be passed to "pytest" or to "run" / "run-cassandra" to control which tests to run:
- To run all tests in a single file, do
pytest test_table.py. - To run a single specific test, do
pytest test_table.py::test_create_table_unsupported_names.
Additional useful pytest options, especially useful for debugging tests:
- -v: show the names of each individual test running instead of just dots.
- -s: show the full output of running tests (by default, pytest captures the test's output and only displays it if a test fails)