Commit Graph

171 Commits

Author SHA1 Message Date
Nadav Har'El
5fbeae9016 cql-pytest: add a couple of default-TTL tests
This patch adds a new cql-pytest test file - test_ttl.py - with
currently just a couple of tests for the "with default_time_to_live"
feature. One is a basic test, and second reproduces issue #9842 -
that "using ttl 0" should override the default time to live, but
doesn't.

The test for #9842, test_default_ttl_0_override, fails on Scylla and
passes on Cassandra, and is marked "xfail".

Refs #9842.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20211227091502.553577-1-nyh@scylladb.com>
2022-01-05 10:15:19 +02:00
Botond Dénes
425c0b0394 test/cql-pytest/nodetool.py: fix take_snapshot() for cassandra
take_snapshot() contained copypasta from flush() for the nodetool
variant.

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20211208110129.141592-1-bdenes@scylladb.com>
2021-12-14 11:15:23 +02:00
Nadav Har'El
41c7b2fb4b test/cql-pytest run: fix inaccurate comment
The code in test/cql-pytest/run.py can start Scylla (or Cassandra, or
Redis, etc.) in a random IP address in 127.*.*.*. We explained in a
comment that 127.0.0.* is used by CCM so we avoid it in case someone
runs both dtest and test.py in parallel on the same machine.

But this observation was not accurate: Although the original CCM did use
only 127.0.0.*, in Scylla's CCM we added in 2017, in commit
00d3ba5562567ab83190dd4580654232f4590962, the ability to run multiple
copies of CCM in parallel; CCM now uses 127.0.*.*, not just 127.0.0.*.
So we need to correct this in the comment.

Luckily, the code doesn't need to change! We already avoided the entire
127.0.*.* for simplicity, not just 127.0.0.*.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20211212151339.1361451-1-nyh@scylladb.com>
2021-12-13 18:12:11 +02:00
Nadav Har'El
c6f2afb93d Merge 'cql3: Allow to skip EQ restricted columns in ORDER BY' from Jan Ciołek
In queries like:
```cql
SELECT * FROM t WHERE p = 0 AND c1 = 0 ORDER BY (c1 ASC, c2 ASC)
```
we can skip the requirement to specify ordering for `c1` column.

The `c1` column is restricted by an `EQ` restriction, so it can have
at most one value anyway, there is no need to sort.

This commit makes it possible to write just:
```cql
SELECT * FROM t WHERE p = 0 AND c1 = 0 ORDER BY (c2 ASC)
```

I reorganized the ordering code, I feel that it's now clearer and easier to understand.
It's possible to only introduce a small change to the existing code, but I feel like it becomes a bit too messy.
I tried it out on the [`orderby_disorder_small`](https://github.com/cvybhu/scylla/commits/orderby_disorder_small) branch.

The diff is a bit messy because I moved all ordering functions to one place,
it's better to read [select_statement.cc](https://github.com/cvybhu/scylla/blob/orderby_disorder/cql3/statements/select_statement.cc#L1495-L1658) lines 1495-1658 directly.

In the new code it would also be trivial to allow specifying columns in any order, we would just have to sort them.
For now I commented out the code needed to do that, because the point of this PR was to fix #2247.
Allowing this would require some more work changing the existing tests.

Fixes: #2247

Closes #9518

* github.com:scylladb/scylla:
  cql-pytest: Enable test for skipping eq restricted columns in order by
  cql3: Allow to skip EQ restricted columns in ORDER BY
  cql3: Add has_eq_restriction_on_column function
  cql3: Reorganize orderings code
2021-12-09 21:11:56 +03:00
Jan Ciolek
13d367dada cql-pytest: Enable test for skipping eq restricted columns in order by
This test was marked as xfail, but now the functionality it tests has been implemented.

In my opinion the expected error message makes no sense, the message was:
"Order by currently only supports the ordering of columns following their declared order in the PRIMARY KEY"
In cases where there was missing restriction on one column.

This has been changed to:
"Unsupported order by relation - column {} doesn't have an ordering or EQ relation."

Because of that I had to modify the test to accept messages from both Scylla and Cassandra.
The expected error message pattern is now "rder by", because that's the largest common part.

Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
2021-12-09 14:59:47 +01:00
Nadav Har'El
e032f92c5c Merge 'api/storage service: validate table names' from Benny Halevy
This series fixes a couple issues around generating and handling of no_such_keyspace and no_such_column_family exceptions.

First, it removes std::throw_with_nested around their throw sites in the respective database::find_* functions.
Fixes #9753

And then, it introduces a `validate_tables` helper in api/storage_service.cc that generates a `bad_param_exception` in order to set the correct http response status if a non-existing table name is provided in the `cf` http request parameter.
Fixes #9754

The series also adds a test for the REST API under test/rest_api that verifies the storage_service enable/disable auto_compaction api and checks the error codes for non-existing keyspace or table.

Test: unit(dev)

Closes #9755

* github.com:scylladb/scylla:
  api: storage_service: add parse_tables
  database: un-nest no_such_keyspace and no_such_column_family exceptions
  database: throw internal error when failing uuid returned by find_uuid
  database: find_uuid: throw no_such_column_family exception if ks/cf were not found
  test: rest_api: add storage_service test
  test: add basic rest api test
  test: cql-pytest: wait for rest api when starting scylla
2021-12-08 16:54:48 +02:00
Benny Halevy
01f2e8b391 test: cql-pytest: wait for rest api when starting scylla
Some of the tests, like nodetool.py, use the scylla REST API.
Add a check_rest_api function that queries http://<node_addr>:10000/
that is served once scylla starts listening on the API port
and call it via run.wait_for_services.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2021-12-08 16:05:32 +02:00
Botond Dénes
0aa4e5e726 test/cql-pytest: mv virtual_tables.py -> test_virtual_tables.py
For consistency with the other tests.

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20211208102108.126492-1-bdenes@scylladb.com>
2021-12-08 12:23:22 +02:00
Jan Ciolek
3ae8752812 cql-pytest: Enable test for UDT with unset values
The test testUDTWithUnsetValues was marked as xfail,
but now the issue has been fixed and we can enable it.

Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
2021-12-03 14:46:21 +01:00
Nadav Har'El
e71131091a cql-pytest: translate Cassandra's tests for user-defined types
This is a translation of Cassandra's CQL unit test source file
validation/entities/UserTypesTest.java into our our cql-pytest
framework.

This test file includes 26 tests for various features and corners of
the user-defined type feature. Two additional tests which were more
involved to translate were dropped with a comment explaining why.

All 26 tests pass on Cassandra, and all but one pass on Scylla:
The test testUDTWithUnsetValues fails on Scylla so marked xfail.
It reproduces a previously-unknown Scylla bug:

  Refs #9671: In some cases, trying to assign an UNSET value into part
              of a UDT is not detected

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20211124074001.708183-1-nyh@scylladb.com>
2021-11-24 10:37:15 +02:00
Botond Dénes
6993a55ff3 test/cql-pytest: add virtual_tables.py
Presence and column check for virtual tables. Where possible (and
simple) more is checked.
2021-11-05 16:26:21 +02:00
Botond Dénes
18f9d329ed test/cql-pytest: nodetool.py: add take_snapshot() 2021-11-05 16:26:01 +02:00
Nadav Har'El
e6d17d8de2 test/cql-pytest: remove "xfail" label on a reproducer for a fixed bug
The two cql-pytest tests:
        test_frozen_collection.py::test_wrong_set_order_in_nested
        test_frozen_collection.py::test_wrong_set_order_in_nested_2

Which used to fail, and therefore marked "xfail", got fixed by commit
5589f348e7 ("cql3: expr: Implement
evaluate(expr::bind_variable). That commit made the handling of bound
variables in prepared statements more rigorous, and in particular
made sure that sets are re-sorted not only if they are at the top
level of the value (as happened in the old code), but also if they
are nested inside some other container. This explains the surprising
fact that we could only reproduce bug with prepared statements, and
only with nested sets - while top-level sets worked correctly.

As the tests no longer failed and the bug tested by them really did
get fixed, in this patch we remove the "xfail" marker from these
tests.

Closes #7856. This issue was really fixed by the aforementioned commit,
but let's close it now.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20211029221731.1113554-1-nyh@scylladb.com>
2021-11-01 10:29:32 +02:00
Piotr Sarna
83722b5563 cql-pytest: add reproducer for deleting based on secondary index
This commit adds a test case for a bug reported by Felipe
<felipemendes@scylladb.com>. The bug involves trying to delete
an entry from a partition based on a secondary index created
on a column which is part of the compound clustering key,
and the unfortunate result is that the whole partition gets wiped.
Cassandra's behavior is in this case correct - deletion based
on a secondary index column is not allowed.

Refs #9495
2021-10-19 08:50:20 +02:00
Nadav Har'El
86e8979ff2 test/alternator, test/cql-pytest: enable specific experimental features
Issue #9467 deprecated the blanket "--experimental" option which we
used to enable all experimental Scylla features for testing, and
suggests that individual experimental features should be enabled
instead.

So this is what we do in this patch for the Scylla-running scripts
in test/alternator and test/cql-pytest: We need to enable UDF for
the CQL tests, and to enable Alternator Streams and Alternator TTL
for the Alternator tests.

Refs #9467

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20211012110312.719654-2-nyh@scylladb.com>
2021-10-15 16:36:35 +03:00
Nadav Har'El
e4bc97349c cql-pytest: XFAILing test was fixed by a Python driver fix
Issue #8203 describes a bug in a long scan which returns a lot of empty
pages (e.g., because most of the results are filtered out). We have two
cql-pytest test cases that reproduced this bug - one for a whole-table
scan and one for a single-partition scan.

It turned out that the bug was not in the Scylla server, but actually in
the Python driver which incorrectly stopped the iteration after an empty
page even though this page did contain the "more pages" flag.

This driver bug was already fixed in the Datastax driver (see
6ed53d9f70,
and in the Scylla fork of the driver:
1d9077d3f4

So in this patch we drop the XFAIL, and if the driver is not new enough
to contain this fix - the test is skipped.

Since our Jenkins machines have the latest Scylla fork of the driver and
it already contains this fix, these tests will not be skipped - and will
run and should pass. Developers who run these tests on their development
machine will see these tests either passing or skipped - depending on
which version of the driver they have installed.

Closes #8203

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20211011113848.698935-1-nyh@scylladb.com>
2021-10-12 10:04:02 +02:00
Nadav Har'El
d1505762df cql-pytest: add to README an example of repeating a test
pytest supports - if the "repeat" extension is installed - a convenient
and efficient way to repeat the same test (or all of them) multiple times.
Since it's very useful, let's document it in cql-pytest/README.md.

By the way, our test.py also has a "--repeat" option, but it can only run
all cql-pytest tests, not just repeat a single small test, and it is also
slower (and arguably, different) because it restarts Scylla instead of
running a test 100 times on the same Scylla.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20211007122146.624210-1-nyh@scylladb.com>
2021-10-07 15:30:41 +03:00
Nadav Har'El
0f8d3ea459 cql-pytest: translate Cassandra's tests for ORDER BY
This is a translation of Cassandra's CQL unit test source file
validation/operations/SelectOrderByTest.java into our our cql-pytest
framework.

This test file includes 17 tests for various features and corners of
SELECT's "ORDER BY" feature. All these tests pass on Cassandra, but
three fail on Scylla and are marked as xfail:

One previously-unknown Scylla bug:

  Refs #9435: SELECT with IN, ORDER BY and function call does not obey
              the ORDER BY

And two new reproducers for already known bugs:

  Refs #2247: ORDER BY should allow skipping equality-restricted
              clustering columns
  Refs #7751: Allow selecting map values and set elements, like in
              Cassandra 4.0

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20211005174140.571056-1-nyh@scylladb.com>
2021-10-06 12:31:38 +03:00
Piotr Sarna
88480ac504 cql-pytest: relax another condition for a failed wasm execution
The previous commit already relaxed the condition for test_fib,
but the same should be done for test_fib_called_on_null
for an identical reason - more than 1 error can be expected
in the case of calling heavily recursive function, and either
fuel exhaustion, or hitting the stack limit, or any other
InvalidRequest exception should be accepted.

Closes #9363
2021-09-23 14:11:02 +03:00
Nadav Har'El
92570ea7d9 cql-pytest: add tests on behavior of empty-string keys
We know (verified by existing tests) that null keys are not allowed -
neither as partition keys nor clustering keys.
In issue #9352 a question was raised of whether an *empty string* is
allowed as as a key on a base table (not a materialized view or index).
The following tests confirm that the current situation is as follows:

1. An empty string is perfectly legal as a clustering key.
2. An empty string is NOT ALLOWED as a partition key - the error
   "Key may not be empty" is reported if this is attempted.
3. If the partition key is compound (multiple partition-key columns)
   then any or all of them may be empty strings.

These tests pass the same on both Cassandra and Scylla, showing that
this bizarre (and undocumented) behavior is identical in both.

Refs #9352.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210922131310.293846-1-nyh@scylladb.com>
2021-09-22 18:55:25 +03:00
Nadav Har'El
a99a774731 cql-pytest: test for secondary-index on empty-string value
When a string column is indexed with a secondary index, the empty value
for this column (an empty string '') is perfectly legal, and should be
indexed as well. This is not the same as an unset (null) value which
isn't indexed.

The following test demonstrates that this case works in Cassandra, but
does not in Scylla (so the test is marked "xfail"). In Scylla, a query
that returns the expected results with ALLOW FILTERING suddenly returns
a different (and wrong) result when an index is added on the table.

This test reproduces issue #9364.

Refs #9364.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210922121510.291826-1-nyh@scylladb.com>
2021-09-22 18:55:25 +03:00
Nadav Har'El
e8493e20cb cql-pytest: test for empty-string as partition key in materialized view
Scylla and Cassandra do not allow an empty string as a partition key,
but a materialized view might "convert" a regular string column into a
partition key, and an empty string is a perfectly valid value for this
column. This can result in a view row which has an empty string as a
partition key. This case works in Cassandra, but doesn't in Scylla (the
row with the empty string as a partition key doesn't appear). The
following test demonstrates this difference between Scylla and Cassandra
(it passes on Cassandra, fails on Scylla, and accordingly marked
"xfail").

Refs #9375.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210922115000.290387-1-nyh@scylladb.com>
2021-09-22 18:55:25 +03:00
Piotr Sarna
dd9d6c081e cql-pytest: relax error conditions for a failed wasm execution
Originally, the expected failure for a recursive invocation
test case was to expect that fuel gets exhausted, but it's also
possible to hit a stack limit first. All errors are equally
expected here as long as the execution is halted, so let's relax
the condition and accept any wasm-related InvalidRequest errors.

Closes #9361
2021-09-20 15:20:52 +03:00
Piotr Sarna
41b94d3cf3 cql-pytest: add wasm-based tests for user-defined functions
A first set of wasm-based test cases is added.
The tests include verifying that supported types work
and that validation of the input wasm is performed.
2021-09-13 19:03:58 +02:00
Avi Kivity
1fd701e709 test: cql-pytest: skip tests depending on timeuuid monotonicity
timeuuid is not monotonic when now() is called on different connections,
so when running tests that depend on that property, we get failures if
using the Scylla driver (which became standard in 729d0fe).

Skip the tests for now, until we figure out what to do. We probably
can't make now() globally monotonic, and there isn't much to gain
by making it monotonic only per connection, since clients are allowed
to switch connections (and even nodes) at will.

Ref #9300

Closes #9323

[avi: committing my own patch to unblock master]
2021-09-12 19:30:40 +03:00
Avi Kivity
c5f52f9d97 schema_tables: don't flush in tests
Flushing schema tables is important for crash recovery (without a flush,
we might have sstables using a new schema before the commitlog entry
noting the schema change has been replayed), but not important for tests
that do not test crash recovery. Avoiding those flushes reduces system,
user, and real time on tests running on a consumer-level SSD.

before:
real	8m51.347s
user	7m5.743s
sys	5m11.185s

after:
real	7m4.249s
user	5m14.085s
sys	2m11.197s

Note real time is higher that user+sys time divided by the number
of hardware threads, indicating that there is still idle time due
to the disk flushing, so more work is needed.

Closes #9319
2021-09-12 11:32:13 +03:00
Dejan Mircevski
6afdc6004c cql3/modification_statement: Replace empty-range check with null check
The empty-range check causes more bugs than it fixes.  Replace it with
an explicit check for =NULL (see #7852).

Fixes #9311.
Fixes #9290.

Tests: unit (dev), cql-pytest on Cassandra 4.0

Signed-off-by: Dejan Mircevski <dejan@scylladb.com>

Closes #9314
2021-09-09 10:56:13 +03:00
Dejan Mircevski
58a9a24ff0 cql3: Allow indexed query to select static columns
We previously forbade selecting a static column when an index is
used.  But Cassandra allows it, so we should, too -- see #8869.

After removing the static-column check, the existing code gets the
correct result without any further changes (though it may read
multiple rows from the same partition).

Fixes #8869.

Tests: unit (dev)

Signed-off-by: Dejan Mircevski <dejan@scylladb.com>

Closes #9307
2021-09-08 08:22:59 +02:00
Dejan Mircevski
1fdaeca7d0 cql3: Reject updates with NULL key values
We were silently ignoring INSERTs with NULL values for primary-key
columns, which Cassandra rejects.  Fix it by rejecting any
modification_statement that would operate on empty partition or
clustering range.

This is the most direct fix, because range and slice are calculated in
one place for all modification statements.  It covers not only NULL
cases, but also impossible restrictions like c>0 AND c<0.
Unfortunately, Cassandra doesn't treat all modification statements
consistently, so this fix cannot fully match its behavior.  We err on
the side of tolerance, accepting some DELETE statements that Cassandra
rejects.  We add a TODO for rejecting such DELETEs later.

Fixes #7852.

Tests: unit (dev), cql-pytest against Cassandra 4.0

Signed-off-by: Dejan Mircevski <dejan@scylladb.com>

Closes #9286
2021-09-05 10:23:28 +03:00
Nadav Har'El
068c4283b7 test/cql-pytest: add tests for some undocumented cases of string types
This patch adds tests for two undocumented (as far as I can tell) corner
cases of CQL's string types:

1. The types "text" and "varchar" are not just similar - they are in
   fact exactly the same type.

2. All CQL string and blob types ("ascii", "text" or "varchar", "blob")
   allow the null character as a valid character inside them. They are
   *not* C strings that get terminated by the first null.

These tests pass on both Cassandra and Scylla, so did not expose any
bug, but having such tests is useful to understand these (so-far)
undocumented behaviors - so we can later document them.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210824225641.194146-1-nyh@scylladb.com>
2021-09-02 15:45:47 +03:00
Avi Kivity
8b59e3a0b1 Merge ' cql3: Demand ALLOW FILTERING for unlimited, sliced partitions ' from Dejan Mircevski
Return the pre- 6773563d3 behavior of demanding ALLOW FILTERING when partition slice is requested but on potentially unlimited number of partitions.  Put it on a flag defaulting to "off" for now.

Fixes #7608; see comments there for justification.

Tests: unit (debug, dev), dtest (cql_additional_test, paging_test)

Signed-off-by: Dejan Mircevski <dejan@scylladb.com>

Closes #9126

* github.com:scylladb/scylla:
  cql3: Demand ALLOW FILTERING for unlimited, sliced partitions
  cql3: Track warnings in prepared_statement
  test: Use ALLOW FILTERING more strictly
  cql3: Add statement_restrictions::to_string
2021-08-31 18:05:26 +03:00
Dejan Mircevski
2f28f68e84 cql3: Demand ALLOW FILTERING for unlimited, sliced partitions
When a query requests a partition slice but doesn't limit the number
of partitions, require that it also says ALLOW FILTERING.  Although
do_filter() isn't invoked for such queries, the performance can still
be unexpectedly slow, and we want to signal that to the user by
demanding they explicitly say ALLOW FILTERING.

Because we now reject queries that worked fine before, existing
applications can break.  Therefore, the behavior is controlled by a
flag currently defaulting to off.  We will default to "on" in the next
Scylla version.

Fixes #7608; see comments there for justification.

Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
2021-08-31 10:45:41 -04:00
Piotr Sarna
88238c7c2a cql-pytest: add test case for UDA with multiple args
A test case for an aggregate which works on multiple parameters
is added.
2021-08-16 19:52:50 +02:00
Piotr Sarna
38c1fd0762 cql-pytest: add a test suite for user-defined aggregates
The test suite now consists of a single user aggregate:
a custom implementation for existing avg() built-in function,
as well as a couple of cases for catching incorrect operations,
like using wrong function signatures or dropping used functions.
2021-08-13 11:16:52 +02:00
Piotr Sarna
5f773d04d2 cql-pytest: add context managers for functions and aggregates
These context managers can be used to create temporary
user-defined functions and user-defined aggregates.
2021-08-13 11:16:52 +02:00
Nadav Har'El
89724533f8 test/cql-pytest: CREATE INDEX IF NOT EXISTS vs. Cassandra
What should the following pair of statements do?

    CREATE INDEX xyz ON tbl(a)
    CREATE INDEX IF NOT EXISTS xyz ON tbl(b)

There are two reasonable choices:
1. An index with the name xyz already exists, so the second command should
   do nothing, because of the "IF NOT EXISTS".
2. The index on tbl(b) does *not* yet exist, so the command should try to
   create it. And when it can't (because the name xyz is already taken),
   it should produce an error message.

Currently, Cassandra went with choice 1, and Scylla went with choice 2.

After some discussions on the mailing list, we agreed that Scylla's
choice is the better one and Cassandra's choice could be considered a
bug: The "IF NOT EXIST" feature is meant to allow idempotent creation of
an index - and not to make it easy to make mistakes without not noticing.
The second command listed above is most likely a mistake by the user,
not anything intentional: The command intended to ensure than an index
on column b exists, but after the silent success of the command, no such
index exists.

So this patch doesn't change any Scylla code (it just adds a comment),
and rather it adds a test which "enshrines" the current behavior.
The test passes on Scylla and fails on Cassandra so we tag it
"cassandra_bug", meaning that we consider this difference to be
intentional and we consider Cassandra's behavior in this case to be wrong.

Fixes #9182.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210811113906.2105644-1-nyh@scylladb.com>
2021-08-11 13:41:58 +02:00
Nadav Har'El
ae51fef57c cql-pytest: add tests for estimated partition count
In issue #9083 a user noted that whereas Cassandra's partition-count
estimation is accurate, Scylla's (rewritten in commit b93cc21) is very
inaccurate. The tests introduce here, which all xfail on Scylla, confirm
this suspicion.

The most important tests are the "simple" tests, involving a workload
which writes N *distinct* partitions and then asks for the estimated
partition count. Cassandra provides accurate estimates, which grow
more accurate with more partitions, so it passes these tests, while
Scylla provides bad estimates and fails them.

Additional tests demonstrate that neither Scylla nor Cassandra
can handle anything beyond the "simple" case of distinct partitions.
Two tests which xfail on both Cassandra and Scylla demonstrate that
if we write the same partitions to multiple sstables - or also delete
partitions - the estimated partition counts will be way off.

Refs #9083

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210726211315.1515856-1-nyh@scylladb.com>
2021-08-05 08:50:19 +02:00
Nadav Har'El
d640998ca8 test/cql-pytest: add test for another ALLOW FILTERING case
In this patch we add another test case for a case where ALLOW FILTERING
should not be required (and Cassandra doesn't require it) but Scylla
does.

This problem was introduced by pull request #9122. The pull request
fixed an incorrect query (see issue #9085) involving both an index and
a multi-column restriction on a compound clustering key - and the fix is
using filtering. However, in one specific case involving a full prefix,
it shouldn't require filtering. This test reproduces this case.

The new test passes on Cassandra (and also theoretically, should pass),
but fails on Scylla - the check_af_optional() call fails because Scylla
made the ALLOW FILTERING mandatory for that case.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210803092046.1677584-1-nyh@scylladb.com>
2021-08-04 15:24:47 +03:00
Dejan Mircevski
debf65e136 cql3: Filter regular-index results on multi-column
When a WHERE clause contains a multi-column restriction and an indexed
regular column, we must filter the results.  It is generally not
possible to craft the index-table query so it fetches only the
matching rows, because that table's clustering key doesn't match up
with the column tuple.

Fixes #9085.

Tests: unit (dev, debug)

Signed-off-by: Dejan Mircevski <dejan@scylladb.com>

Closes #9122
2021-08-02 14:15:43 +03:00
Pavel Solodovnikov
b1a3b59a08 test: test_materialized_view: test_mv_select_stmt_bound_values: improve error handling
Restrict expected exception message to filter only relevant exception,
matching both for scylla and cassandra.

For example, the former has this message:

    Cannot use query parameters in CREATE MATERIALIZED VIEW statements

While the latter throws this:

    Bind variables are not allowed in CREATE MATERIALIZED VIEW statements

Also, place cleanup code in try-finally clause.

Tests: cql-pytest:test_materialized_view.py(dev)

Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
Message-Id: <20210802083912.229886-1-pa.solodovnikov@scylladb.com>
2021-08-02 11:49:50 +03:00
Nadav Har'El
e8fe1817df cql-pytest: translate Cassandra's tests for timestamps
This is a translation of Cassandra's CQL unit test source file
validation/entities/TimestampTest.java into our our cql-pytest framework.

This test file checks has a few tests (8) on various features of
cell timestamps. All these tests pass on Cassandra and on Scylla - i.e.,
these tests no new Scylla bug was detected :-)

Two of the new tests are very slow (6 seconds each) and check a trivial
feature that was already checked elsewhere more efficiently (the fact
that TTL expiration works), so I marked them "skip" after verifying they
really pass.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210801142738.1633126-1-nyh@scylladb.com>
2021-08-02 09:25:49 +02:00
Pavel Solodovnikov
d07f681a95 test: test_non_deterministic_functions: add lwt to test cases names
The tests are related to LWT so add the corresponding prefix
to all the tests cases to emphasize that.

Tests: cql-pytest(dev)

Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
Message-Id: <20210801131820.164480-1-pa.solodovnikov@scylladb.com>
2021-08-01 16:23:30 +03:00
Avi Kivity
48860b135a Merge "cql3: fix current*() functions to be non-deterministic" from Pavel S
"
Previously, the following functions were
incorrectly marked as pure, meaning that the
function is executed at "prepare" step:

* `currenttimestamp()`
* `currenttime()`
* `currentdate()`
* `currenttimeuuid()`

For functions that possibly depend on timing and random seed,
this is clearly a bug. Cassandra doesn't have a notion of pure
functions, so they are lazily evaluated.

Make Scylla to match Cassandra behavior for these functions.

Add a unit-test for a fix (excluding `currentdate()` function,
because there is no way to use synthetic clock with query
processor and sleeping for a whole day to demonstrate correct
behavior is clearly not an option).

Also, extend the cql-pytest for #8604 since there are now more
non-deterministic CQL functions, they are all subject to the test
now.

Fixes: #8816
"

* 'timeuuid_function_pure_annotation_v3' of https://github.com/ManManson/scylla:
  test: test_non_deterministic_functions: test more non-pure functions
  cql3: change `current*()` CQL functions to be non-pure
2021-08-01 12:35:36 +03:00
Pavel Solodovnikov
a130921120 test: test_non_deterministic_functions: test more non-pure functions
Check that all existing non-pure functions (except for
`currentdate()`) work correctly with or without prepared
statements.

Tests: cql-pytest/test_non_deterministic_functions.py(dev)

Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
2021-08-01 12:18:26 +03:00
Pavel Solodovnikov
1ca7825cf6 test: add a test checking that bind markers within MVs SELECT statement don't lead to a crash
The request should fail with `InvalidRequest` exception and shouldn't
crash the database.

Don't check for actual error messages, because they are different
between Scylla and Cassandra.

The former has this message:

        Cannot use query parameters in CREATE MATERIALIZED VIEW statements

While the latter throws this:

        Bind variables are not allowed in CREATE MATERIALIZED VIEW statements

Tests: cql-pytest/test_materialized_view.py(scylla dev, cassandra trunk)

Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
2021-07-30 17:57:24 +03:00
Piotr Sarna
1c7af8d46f cql-pytest: adjust a test case for Cassandra 4
One of the test cases stopped working against Cassandra 4, but that's
just because it returns a slightly different error type.
The test case is adjusted to work on both Scylla and new Cassandra.

Message-Id: <222a7f63a3e9739c6fc646173306fcdb3da25890.1627655555.git.sarna@scylladb.com>
2021-07-30 17:36:23 +03:00
Avi Kivity
0876248c2b Merge "cql3: cache function calls evaluation for non-deterministic functions" from Pavel S
"
`function_call` AST nodes are created for each function
with side effects in a CQL query, i.e. non-deterministic
functions (`uuid()`, `now()` and some others timeuuid-related).

These nodes are evaluated either when a query itself is executed
or query restrictions are computed (e.g. partition/clustering
key ranges for LWT requests).

We need to cache the calls since otherwise when handling a
`bounce_to_shard` request for an LWT query, we can possibly
enter an infinite bouncing loop (in case a function is used
to calculate partition key ranges for a query), since the
results can be different each time.

Furthermore, we don't support bouncing more than one time.
Returning `bounce_to_shard` message more than one time
will result in a crash.

Caching works only for LWT statements and only for the function
calls that affect partition key range computation for the query.

`variable_specifications` class is renamed to `prepare_context`
and generalized to record information about each `function_call`
AST node and modify them, as needed:
* Check whether a given function call is a part of partition key
  statement restriction.
* Assign ids for caching if above is true and the call is a part
  of an LWT statement.

There is no need to include any kind of statement identifier
in the cache key since `query_options` (which holds the cache)
is limited to a single statement, anyway.

Function calls are indexed by the order in which they appear
within a statement while parsing. There is no need to
include any kind of statement identifier to the cache key
since `query_options` (which holds the cache) is limited
to a single statement, anyway.

Note that `function_call::raw` AST nodes are not created
for selection clauses of a SELECT statement hence they
can only accept only one of the following things as parameters:
* Other function calls.
* Literal values.
* Parameter markers.

In other words, only parameters that can be immediately reduced
to a byte buffer are allowed and we don't need to handle
database inputs to non-pure functions separately since they
are not possible in this context. Anyhow, we don't even have
a single non-pure function that accepts arguments, so precautions
are not needed at the moment.

Add a test written in `cql-pytest` framework to verify
that both prepared and unprepared lwt statements handle
`bounce_to_shard` messages correctly in such scenario.

Fixes: #8604

Tests: unit(dev, debug)

NOTE: the patchset uses `query_options` as a container for
cached values. This doesn't look clean and `service::query_state`
seems to be a better place to store them. But it's not
forwarded to most of the CQL code and would mean that a huge number
of places would have to be amended.
The series presents a trade-off to avoid forwarding `query_state`
everywhere (but maybe it's the thing that needs to be done, nonetheless).
"

* 'lwt_bounce_to_shard_cached_fn_v6' of https://github.com/ManManson/scylla:
  cql-pytest: add a test for non-pure CQL functions
  cql3: cache function calls evaluation for non-deterministic functions
  cql3: rename `variable_specifications` to `prepare_context`
2021-07-30 14:21:11 +03:00
Pavel Solodovnikov
eaf70df203 cql-pytest: add a test for non-pure CQL functions
Introduce a test using `cql-pytest` framework to assert that
both prepared an unprepared LWT statements (insert with
`IF NOT EXISTS`) with a non-deterministic function call
work correctly in case its evaluation affects partition
key range computation (hence the choice of `cas_shard()`
for lwt query).

Tests: cql-pytest/test_non_deterministic_functions.py

Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
2021-07-30 01:22:50 +03:00
Nadav Har'El
8030461a2c cql-pytest: translate Cassandra's misc. type tests
This is a translation of Cassandra's CQL unit test source file
validation/entities/TypeTest.java into our our cql-pytest framework.

This is a tiny test file, with only four test which apparently didn't
find their place in other source files. All four tests pass on Cassandra,
and all but one pass on Scylla - the test marked xfail discovered one
previously-unknown incompatibility with Cassandra:

Refs #9082: DROP TYPE IF EXISTS shouldn't fail on non-existent keyspace

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210726140934.1479443-1-nyh@scylladb.com>
2021-07-27 08:28:16 +03:00
Nadav Har'El
b503ec36c2 cql-pytest: translate Cassandra's tests for tuples
This is a translation of Cassandra's CQL unit test source file
validation/entities/TupleTypeTest.java into our our cql-pytest framework.

This test file checks has a few tests on various features of tuples.
Unfortunately, some of the tests could not be easily translated into
Python so were left commented out: Some tests try to send invalid input
to the server which the Python driver "helpfully" forbids; Two tests
used an external testing library "QuickTheories" and are the only two
tests in the Cassandra test suite to use this library - so it's not
a worthwhile to translate it to Python.

11 tests remain, all of them pass on Cassandra, and just one fails on
Scylla (so marked xfail for now), reproducing one known issue:

Refs #7735: CQL parser missing support for Cassandra 3.10's new "+=" syntax
Actually, += is not supposed to be supported on tuple columns anyway, but
should print the appropriate error - not the syntax error we get now as
the "+=" feature is not supported at all.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210722201900.1442391-1-nyh@scylladb.com>
2021-07-26 08:20:12 +03:00