Commit Graph

440 Commits

Author SHA1 Message Date
Jan Ciolek
5e5e4c5323 cql-pytest/test_lwt: test LWT update with empty partition range
Add a test case which performs an LWT UPDATE, but the partition key
has 0 possible values, because it's supposed to be equal to two
different values.
Such queries used to cause problems in the past.

Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
2023-03-09 15:43:24 +01:00
Nadav Har'El
a4a318f394 cql: USING TTL 0 means unlimited, not default TTL
Our documentation states that writing an item with "USING TTL 0" means it
should never expire. This should be true even if the table has a default
TTL. But Scylla mistakenly handled "USING TTL 0" exactly like having no
USING TTL at all (i.e., it took the default TTL, instead of unlimited).
We had two xfailing tests demonstrating that Scylla's behavior in this
is different from Cassandra. Scylla's behavior in this case was also
undocumented.

By the way, Cassandra used to have the same bug (CASSANDRA-11207) but
it was fixed already in 2016 (Cassandra 3.6).

So in this patch we fix Scylla's "USING TTL 0" behavior to match the
documentation and Cassandra's behavior since 2016. One xfailing test
starts to pass and the second test passes this bug and fails on a
different one. This patch also adds a third test for "USING TTL ?"
with UNSET_VALUE - it behaves, on both Scylla and Cassandra, like a
missing "USING TTL".

The origin of this bug was that after parsing the statement, we saved
the USING TTL in an integer, and used 0 for the case of no USING TTL
given. This meant that we couldn't tell if we have USING TTL 0 or
no USING TTL at all. This patch uses an std::optional so we can tell
the case of a missing USING TTL from the case of USING TTL 0.

Fixes #6447

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #13079
2023-03-08 16:18:23 +02:00
Jan Ciolek
0417c48bdc cql-pytest: test unset value in UPDATE and LWT UPDATE
Add a test which performs an UPDATE and
tries to pass an UNSET_VALUE as a value
for the primary key.

There is also an LWT variant of this test
that tries to set an UNSET_VALUE
in the IF condition.

These two tests are analogous to
test_insert_update_where and
test_insert_update_where_lwt,
but use an UPDATE instead of INSERT.

It's useful to test UPDATE as well as INSERT.
When I was developing a fix for #13001
I initially added the condition for unset value
inside insert_statement, but this didn't handle
update statements. These two tests allowed me
to see that UPDATE still causes a crash.

Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>

Closes #13058
2023-03-08 10:39:26 +02:00
Nadav Har'El
ef50e4022c test: drop our "pytest" wrapper script
When Fedora 37 came out, we discovered that its "pytest" script started
to run Python with the "-s" option, which caused problems for packages
installed personally via pip. We fixed this by adding our own wrapper
script test/pytest.

But this bug (https://bugzilla.redhat.com/show_bug.cgi?id=2152171) was
already fixed in Fedora 37, and the new version already reached our
dbuild. So we no longer need this wrapper script. Let's remove it.

Fixes #12412

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #13083
2023-03-08 07:31:37 +02:00
Nadav Har'El
f05ea80fb5 test/cql-pytest: remove unused async marker
One test in test/cql-pytest/test_batch.py accidentally had the asyncio
marker, despite not using any async features. Remove it. The test still
runs fine.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #13002
2023-03-07 14:33:34 +02:00
Wojciech Mitros
d4851ccae7 treewide: rename the "xwasm" UDF language to "wasm"
When the WASM UDFs were first introduced, the LANGUAGE required in
the CQL statements to use them was "xwasm", because the ABI for the
UDFs was still not specified and changes to it could be backwards
incompatible.
Now, the ABI is stabilized, but if backwards incompatible changes
are made in the future, we will add a new ABI version for them, so
the name "xwasm" is no longer needed and we can finally
change it to "wasm".

Closes #13089
2023-03-07 10:21:11 +02:00
Alejo Sanchez
eaed778f4a test/cql-pytest: print driver version
Print driver version for cql-pytest tests.

Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>

Closes #12840
2023-03-06 11:31:26 +02:00
Botond Dénes
92fde47261 Merge 'test/cql-pytest - aggregation tests' from Nadav Har'El
This small series reorganizes the existing functional tests for aggregation (min, max, count) and adds additional tests for sum reproducing the strange (but Cassandra-compatible) behavior described in issue #13027.

Closes #13038

* github.com:scylladb/scylladb:
  cql-pytest: add tests for sum() aggregate
  test/cql-pytest: move aggregation tests to one file
2023-03-01 11:02:08 +02:00
Nadav Har'El
363f326d49 test/cql-pytest: test for CLUSTERING ORDER BY verification in MV
Since commit 73e258fc34, Scylla has partial
verification for the CLUSTERING ORDER BY clause in CREATE MATERIALIZED
VIEW. Specifically, invalid column names are rejected. But for reasons
explained in issue #12936 and in the test in this patch, Cassandra
demands that if CLUSTERING ORDER BY appears it must list all the
clustering columns, with no duplicates, and do so in the right order.

This patch replaces an existing test which suggested it is fine
(an extention over Cassandra) to accept a partial list of clustering
columns, by a test that verifies that such a partial list, or an
incorrectly-ordered list, or list with duplicates, should be rejected.
The new test fails on Scylla, and passes on Cassandra, so marked as xfail.

Refs #12936.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12938
2023-03-01 08:02:39 +02:00
Nadav Har'El
7dc54771e1 test/cql-pytest: allow "run-cassandra" without building Scylla
Before this patch, all scripts which use test/cql-pytest/run.py
looked for the Scylla executable as their first step. This is usually
the right thing to do, except in two cases where Scylla is *not* needed:

1. The script test/cql-pytest/run-cassandra.
2. The script test/alternator/run with the "--aws" option.

So in this patch we change run.py to only look for Scylla when actually
needed (the find_scylla() function is called). In both cases mentioned
above, find_scylla() will never get called and the script can work even
if Scylla was never built.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #13010
2023-03-01 07:54:19 +02:00
Wojciech Mitros
6d2e785b5c docs: update wasm.md
The WASM UDF implementation has changed since the last time the docs
were written. In particular, the Rust helper library has been
released, and using it should be the recommended method.
Some decisions that were only experimental at the start, were also
"set in stone", so we should refer to them as such.

The docs also contain some code examples. This patch adds tests for
these examples to make sure that they are not wrong and misleading.

Closes #12941
2023-02-28 20:59:25 +02:00
Nadav Har'El
130c090251 cql-pytest: add tests for sum() aggregate
This patch adds regression tests for the strange (but Cassandra-compatible)
behavior described in issue #13027 - that sum of no results returns 0
(not null or nothing), and if also asking for p, we get a null there too.

Refs #13027.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2023-02-28 15:35:21 +02:00
Nadav Har'El
e1f97715eb test/cql-pytest: move aggregation tests to one file
We had separate test files test_minmax.py and test_count.py but the
separate was artificial (and test_count.py even had one test using
min()). Now I that want to add another test for sum(), I don't know
where to put it. So in this patch I combine test_minmax.py and
test_count.py into one test file - test_aggregate.py, and we can
later add sum() tests in the same file.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2023-02-28 14:39:04 +02:00
Kefu Chai
4f3bc915a6 cql-pytest: remove duplicated words in README.md
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes #13005
2023-02-27 17:28:32 +02:00
Nadav Har'El
3b32440993 test/cql-pytest: add regression test for UNSET key in insert
Recently, we overhauled the error handling of UNSET_VALUE in various
places where it is not allowed. This patch adds two more regression
tests for this error handling. Both tests pass on Scylla today, pass
on Cassandra, but fail on earlier Scylla (e.g., I tested 5.1.5):

The first test does INSERT into clustering key UNSET_VALUE.
An UNSET_VALUE is designed to skip part of the write - not an entire
write - so this attempt should fail - not silently be skipped.
The write indeed fails with an error on Cassandra, and on recent
Scylla, but silently did nothing in older Scylla which leads this
test to fail there.

The second test does the same thing with LWT (adding an "IF NOT EXISTS")
added to the insert. Scylla's failure here was even more spectacular -
it crashed (as reported in issue #13001) instead of silently skipping
the right. The test passes on Scylla today and on Cassandra, which
both report the failure cleanly.

Refs #13001.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #13007
2023-02-27 17:20:22 +02:00
Nadav Har'El
73e258fc34 materialized views: verify CLUSTERING ORDER BY clause
Cassandra is very strict in the CLUSTERING ORDER BY clause which it
allows when creating a materialized view - if it appears, it must
list all the clustering columns of the view. Scylla is less strict -
a subset of the clustering columns may be specified. But Scylla was
*too* lenient - a user could specify non-clustering columns and even
non-existent columns and Scylla would not fail the MV creation.
This patch fixes that - with it MV creation fails if anything besides
clustering columns are listed on CLUSTERING ORDER BY.

An xfailing test we had for this case no longer fails after this
patch so its xfail mark is removed. We also add a few more corner
cases to the tests.

This patch also fixs one C++ test which had exactly the error that this
patch detects - the test author tried to use the partition key, instead
of the clustering key, in CLUSTERING ORDER BY (this error had no effect
because the specified order, "asc", was the default anyway).

Fixes #10767

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12885
2023-02-27 15:09:42 +02:00
Kefu Chai
79d2eb1607 cql3: functions: validate arguments for 'token()' also
since "token()" computes the token for a given partition key,
if we pass the key of the wrong type, it should reject.

in this change,

* we validate the keys before returning the "token()" function.
* drop the "xfail" decorator from two of the tests. they pass
  now after this fix.
* change the tests which previously passed the wrong number of
  arguments containing null to "token()" and expect it to return
  null, so they verify that "token()" should reject these
  arguments with the expected error message.

Fixes #10448
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes #12991
2023-02-26 19:01:58 +02:00
Pavel Emelyanov
3f88d3af62 Merge 'test_shed_too_large_request fix: disable compression' from Gusev Petr
The test relies on exact request size, this doesn't work if compression is applied. The driver enables compression only if both the server and the client agree on the codec to use. If compression package
(e.g. lz4) is not installed, the compression
is not used.

The trick with locally_supported_compressions is needed since I couldn't find any standard means to disable compression other than the compression flag
on the cluster object, which seemed too broad.

fixes: #12836

Closes #12854

* github.com:scylladb/scylladb:
  test_shed_too_large_request: clarify the comments
  test_shed_too_large_request: use smaller test string
  test_shed_too_large_request fix: disable compression
2023-02-21 10:35:59 +03:00
Nadav Har'El
328cdb2124 cql-pytest: translate Cassandra's tests for compact tables
This is a translation of Cassandra's CQL unit test source file
validation/operations/CompactStorageTest.java into our cql-pytest
framework.

This very large test file includes 86 tests for various types of
operations and corner cases of WITH COMPACT STORAGE tables.

All 86 tests pass on Cassandra (except one using a deprecated feature
that needs to be specially enabled). 30 of the tests fail on Scylla
reproducing 7 already-known Scylla issues and 7 previously-unknown issues:

Already known issues:

Refs #3882: Support "ALTER TABLE DROP COMPACT STORAGE"
Refs #4244: Add support for mixing token, multi- and single-column
            restrictions
Refs #5361: LIMIT doesn't work when using GROUP BY
Refs #5362: LIMIT is not doing it right when using GROUP BY
Refs #5363: PER PARTITION LIMIT doesn't work right when using GROUP BY
Refs #7735: CQL parser missing support for Cassandra 3.10's new "+=" syntax
Refs #8627: Cleanly reject updates with indexed values where value > 64k

New issues:

Refs #12471: Range deletions on COMPACT STORAGE is not supported
Refs #12474: DELETE prints misleading error message suggesting
             ALLOW FILTERING would work
Refs #12477: Combination of COUNT with GROUP BY is different from
             Cassandra in case of no matches
Refs #12479: SELECT DISTINCT should refuse GROUP BY with clustering column
Refs #12526: Support filtering on COMPACT tables
Refs #12749: Unsupported empty clustering key in COMPACT table
Refs #12815: Hidden column "value" in compact table isn't completely hidden

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12816
2023-02-20 12:46:15 +01:00
Nadav Har'El
02682aa40d test/cql-pytest: add reproducer for ALLOW FILTERING bug
This patch adds a reproducer for the bug described in issue #7964 -
The restriction `where k=1 and c=2` (when k,c are the key columns)
returns (at most) a single row so doesn't need ALLOW FILTERING,
but if we add a third restriction, say `v=2`, this still processes
at most a single row so doesn't need ALLOW FILTERING - and both
Scylla and Cassandra get it wrong - so it's marked with both xfail
and cassandra_bug.

The patch also adds another test that for longer partition slices,
e.g., `where k=1 and c>2`, although the slice itself doesn't need
filtering, if we add `v=2` here we suddenly do need ALLOW FILTERING,
because the slice itself may be a large number of rows, and adding
`v=2` may restrict it to just a few results. This test passes
on both Scylla and Cassandra.

Issue #7964 mentioned these scenarios and even had some example code,
but we never added it to the test suite, so we finally do it now.

Refs #7964

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12850
2023-02-16 19:05:48 +02:00
Nadav Har'El
27ea908c69 test/cql-pytest: regression test for old secondary-index bug
This patch adds a cql-pytest test for an old secondary-index bug
that was described three years ago in issue #5823. cql-pytest makes
it easy to run the same test against different versions of Scylla,
and it was used to check that the bug existed in Scylla 2.3.0 but
was gone by 2.3.5, and also not present in master or in 2021.1.

A bit about the bug itself:

A secondary index is useful for equality restrictions (a=2) but can't be
used for inequality restrictions (a>=2). In Scylla 3.2.0 we used to have a
bug that because the restriction a>=2 couldn't be used through the index,
it was ignored completely. This is of course a mistake.

Refs #5823

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12856
2023-02-16 19:05:48 +02:00
Petr Gusev
b37eee26e1 test_shed_too_large_request: clarify the comments 2023-02-15 17:18:17 +04:00
Petr Gusev
4328f52242 test_shed_too_large_request: use smaller test string
There was a vague comment about CI using
larger limits for shedding. This turned out
to be false, and the real reason of different
limits is that Scylla handles the -m
command line option differently in
debug and release builds.
Debug builds use the default memory allocator
and the value of -m Scylla option
is given to each shard. In release builds
memory is evenly distributed between shards.

To accommodate for this we read the current
memory limit from Scylla metrics.
The helper class ScyllaMetrics was introduced to
handle metrics parsing logic. It can
potentially be reused for dealing with
metrics in other tests.
2023-02-15 17:18:10 +04:00
Nadav Har'El
2d6c53c047 test/cql-pytest: reproduce bug in static-column index lookup
This patch adds a reproducer to a static-column index lookup bug
described in issue #12829: The restriction `where pk=0 and s=1 and c=3`
where pk,c are the primary key and s is an indexed static column,
results in an internal error: "clustering column id 2 >= 2".

Unfortunately, because on_internal_error() crashes Scylla in debug
mode, we need to mark this failing test with skip instead of xfail.

Refs #12829

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12852
2023-02-15 12:23:36 +02:00
Petr Gusev
1f850374fa test_shed_too_large_request fix: disable compression
The test relies on exact request size, this doesn't
work if compression is applied. The driver enables
compression only if both the server and the client
agree on the codec to use. If compression package
(e.g. lz4) is not installed, the compression
is not used.

The trick with locally_supported_compressions is needed
since I couldn't find any standard means to disable
compression other than the compression flag
on the cluster object, which seemed too broad.

Fixes #12836
2023-02-15 11:55:49 +04:00
Nadav Har'El
c0114d8b02 test/cql-pytest: test another case of ALLOW FILTERING
In issue #12828 it was noted that Scylla requires ALLOW FILTERING
for `where b=1 and c=1` where b is an indexed static column and
c is a clustering key, and it was suggested that this is a bug.

This patch adds a test that confirms that both Scylla and Cassandra
require ALLOW FILTERING in this case. We explain in a comment that
this requirement is expected (i.e., it's not a bug), as the `b=1`
may match a huge number of rows, and the `c=1` may further match just
a few of those - i.e., it is filtering.

This test is virtually identical to the test we already had for
`where a=1 and c=1` - when `a` is an indexed regular column.
There too, the ALLOW FILTERING is required.

Closes #12828 as "not a bug".

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12848
2023-02-15 08:43:19 +02:00
Nadav Har'El
310638e84d Merge 'wasm: deserialize counters as integers' from Wojciech Mitros
Currently, because serialize_visitor::operator() is not implemented for counters, we cannot convert a counter returned by a WASM UDF to bytes when returning from wasm::run_script().

We could disallow using counters as WASM UDF return types, but an easier solution which we're already using in Lua UDFs is treating the returned counters as 64-bit integers when deserializing. This patch implements the latter approach and adds a test for it.

Closes #12806

* github.com:scylladb/scylladb:
  wasm udf: deserialize counters as integers
  test_wasm.py: add utility function for reading WASM UDF saved in files
2023-02-13 19:24:11 +02:00
Nadav Har'El
6a45881d22 Merge 'functions: handle replacing UDFs used in UDAs' from Wojciech Mitros
This patch is based on #12681, only last 3 commits are relevant.

As described in #12709, currently, when a UDF used in a UDA is replaced, the UDA is not updated until the whole node is restarted.

This patch fixes the issue by updating all affected UDAs when a UDF is replaced.
Additionally, it includes a few convenience changes

Closes #12710

* github.com:scylladb/scylladb:
  uda: change the UDF used in a UDA if it's replaced
  functions: add helper same_signature method
  uda: return aggregate functions as shared pointers
2023-02-13 16:30:24 +02:00
Nadav Har'El
efed973dd3 Merge 'cql3: convert LWT IF clause to expressions' from Avi Kivity
LWT `IF` (column_condition) duplicates the expression prepare and evaluation code. Annoyingly,
LWT IF semantics are a little different than the rest of CQL: a NULL equals NULL, whereas usually
NULL = NULL evaluates to NULL.

This series converts `IF` prepare and evaluate to use the standard expression code. We employ
expression rewriting to adjust for the slightly different semantics.

In a few places, we adjust LWT semantics to harmonize them with the rest of CQL. These are pointed
out in their own separate patches so the changes don't get lost in the flood.

Closes #12356

* github.com:scylladb/scylladb:
  cql3: lwt: move IF clause expression construction to grammar
  cql3: column_condition: evaluate column_condition as a single expression
  cql3: lwt: allow negative list indexes in IF clause
  cql3: lwt: do not short-circuit col[NULL] in IF clause
  cql3: column_condition: convert _column to an expression
  cql3: expr: generalize evaluation of subscript expressions
  cql3: expr: introduce adjust_for_collection_as_maps()
  cql3: update_parameters: use evaluation_inputs compatible row prefetch
  cql3: expr: protect extract_column_value() from partial clustering keys
  cql3: expr: extract extract_column_value() from evaluation machinery
  cql3: selection: introduce selection_from_partition_slice
  cql3: expr: move check for ordering on duration types from restrictions to prepare
  cql3: expr: remove restrictions oper_is_slice() in favor of expr::is_slice()
  cql3: column_condition: optimize LIKE with constant pattern after preparing
  cql3: expr: add optimizer for LIKE with constant pattern
  test: lib: add helper to evaluate an expression with bind variables but no table
  cql3: column_condition: make the left-hand-side part of column_condition::raw
  cql3: lwt: relax constraints on map subscripts and LIKE patterns
  cql3: expr: fix search_and_replace() for subscripts
  cql3: expr: fix function evaluation with NULL inputs
  cql3: expr: add LWT IF clause variants of binary operators
  cql3: expr: change evaluate_binop_sides to return more NULL information
2023-02-13 16:30:24 +02:00
Nadav Har'El
25610c81fb test/cql-pytest: another reproducer for index+limit+filtering bug
This patch adds yet another reproducer for issue #10649, where a
the combination of filtering and LIMIT returns fewer results when
a secondary index is added to the table.

Whereas the previous tests we had for this issue involved a regular
(global) index, the new test uses a local index (a Scylla-only feature).
It shows that the same bug exists also for local indexes, as noticed
by a user in #12766.

Refs #10649
Refs #12766

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12783
2023-02-13 16:30:24 +02:00
Nadav Har'El
ecfcb93ef5 test/cql-pytest: regression test for old bug of misused index
Issue #7659, which we solved long ago, was about a query which included
a non-EQ restriction and wrongly picked up one of the indexes. It had
a short C++ regression test, but here we add a more elaborate Python
test for the same bug. The advantages of the Python test are:

1. The Python test can be run against any version of Scylla (e.g., to
   whether a certain version contains a backport of the fix).

2. The Python test reproduces not only a "benign" query error, but also
   an assertion-failed crash which happened when the non-EQ restriction
   was an "IN".

3. The Python test reproduces the same bug not just for a regular
   index, but also a local index.

I checked that, as expected, these tests pass on master, but fail
(and crash Scylla) in old branches before the fix for #7659.

Refs #7659.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12797
2023-02-13 16:30:24 +02:00
Wojciech Mitros
b25ee62f75 wasm udf: deserialize counters as integers
Currently, because serialize_visitor::operator() is not implemented
for counters, we cannot convert a counter returned by a WASM UDF
to bytes when returning from wasm::run_script().

We could disallow using counters as WASM UDF return types, but an
easier solution which we're already using in Lua UDFs is treating
the returned counters as 64-bit integers when deserializing. This
patch implements the latter approach and adds a test for it.
2023-02-13 14:24:20 +01:00
Wojciech Mitros
3b8bf1ae3a test_wasm.py: add utility function for reading WASM UDF saved in files
Currently, we're repeating the same os.path, open, read, replace
each time we read a WASM UDF from a file.

To reduce code bloat, this patch adds a utility function
"read_function_from_file" that finds the file and reads it given
a function name and an optional new name, for cases when we want
to use a different name in cql (mostly for unique_names).
2023-02-13 14:24:20 +01:00
Avi Kivity
31ee13c0c9 cql3: expr: move check for ordering on duration types from restrictions to prepare
Both LWT IF clause and SELECT WHERE clause check that a duration type
isn't used in an ordered comparison, since duration types are unordered
(is 1mo more or less than 30d?). As a first step towards centralizing this
check, move the check from restrictions into prepare. When LWT starts using
prepare, the duplication will be removed.

The error message was changed: the word "slice" is an internal term, and
a comparison does not necessarily have to be in a restriction (which is
also an internal term).

Tests were adjusted.
2023-02-12 17:17:01 +02:00
Nadav Har'El
10ca08e8ac Merge 'Sequence CDC preimage select with Paxos learn write' from Kamil Braun
`paxos_response_handler::learn_decision` was calling
`cdc_service::augment_mutation_call` concurrently with
`storage_proxy::mutate_internal`. `augment_mutation_call` was selecting
rows from the base table in order to create the preimage, while
`mutate_internal` was writing rows to the table. It was therefore
possible for the preimage to observe the update that it accompanied,
which doesn't make any sense, because the preimage is supposed to show
the state before the update.

Fix this by performing the operations sequentially. We can still perform
the CDC mutation write concurrently with the base mutation write.

`cdc_with_lwt_test` was sometimes failing in debug mode due to this bug
and was marked flaky. Unmark it.

Also fix a comment in `cdc_with_lwt_test`.

Fixes #12098

Closes #12768

* github.com:scylladb/scylladb:
  test/cql-pytest: test_cdc: regression test for #12098
  test/cql: cdc_with_lwt_test: fix comment
  service: storage_proxy: sequence CDC preimage select with Paxos learn
2023-02-12 13:28:34 +02:00
Kamil Braun
540f6d9b78 test/cql-pytest: test_cdc: regression test for #12098
Perform multiple LWT inserts to different keys ensuring none of them
observes a preimage.

On my machine this test reproduces the problem more than 50% of the time
in debug mode.
2023-02-10 14:35:49 +01:00
Botond Dénes
1c333e2102 Merge 'Transport server error handling fixes' from Gusev Petr
CQL transport sever error handling fixes and improvements:
  * log failed requests with `DEBUG` level for easier debugging;
  * in case of unhandled errors, deliver them to the client as `SERVER_ERROR`'s
  * fix for `protocol_error`'s in case of shedded big requests;
  * explicit tests have been written for the error handling problems above.

Closes #11949

* github.com:scylladb/scylladb:
  transport server: fix "request size too large" handling
  transport server: log failed requests with debug level
  transport server: fix unexpected server errors handling
  transport server: log client errors with debug level
2023-02-09 09:02:22 +02:00
Petr Gusev
3263523b54 transport server: fix "request size too large" handling
Calling _read_buf.close() doesn't imply eof(), some data
may have already been read into kernel or client buffers
and will be returned next time read() is called.
When the _server._max_request_size limit was exceeded
and the _read_buf was closed, the process_request method
finished and we started processing the next request in
connection::process. The unread data from _read_buf was
treated as the header of the next request frame, resulting
in "Invalid or unsupported protocol version" error.

The existing test_shed_too_large_request was adjusted.
It was originally written with the assumption that the data
of a large query would simply be dropped from the socket
and the connection could be used to handle the
next requests. This behaviour was changed in scylladb#8800,
now the connection is closed on the Scylla side and
can no longer be used. To check there are no errors
in this case, we use Scylla metrics, getting them
from the Scylla Prometheus API.
2023-02-08 00:07:08 +04:00
Petr Gusev
a4cf509c3d transport server: fix unexpected server errors handling
If request processing ended with an error, it is worth
sending the error to the client through
make_error/write_response. Previously in this case we
just wrote a message to the log and didn't handle the
client connection in any way. As a result, the only
thing the client got in this case was timeout error.

A new test_batch_with_error is added. It is quite
difficult to reproduce error condition in a test,
so we use error injection instead. Passing injection_key
in the body of the request ensures that the exception
will be thrown only for this test request and
will not affect other requests that
the driver may send in the background.

Closes: scylladb#12104
2023-02-08 00:07:02 +04:00
Wojciech Mitros
02bfac0c66 uda: change the UDF used in a UDA if it's replaced
Currently, if a UDA uses a UDF that's being replaced,
the UDA will still keep using the old UDF until the
node is restarted.
This patch fixes this behavior by checking all UDAs
when replacing a UDF and updating them if necessary.

Fixes #12709
2023-02-07 12:17:52 +01:00
Nadav Har'El
3ba011c2be cql: fix empty aggregation, and add more tests
This patch fixes #12475, where an aggregation (e.g., COUNT(*), MIN(v))
of absolutely no partitions (e.g., "WHERE p = null" or "WHERE p in ()")
resulted in an internal error instead of the "zero" result that each
aggregator expects (e.g., 0 for COUNT, null for MIN).

The problem is that normally our aggregator forwarder picks the nodes
which hold the relevant partition(s), forwards the request to each of
them, and then combines these results. When there are no partitions,
the query is sent to no node, and we end up with an empty result set
instead of the "zero" results. So in this patch we recognize this
case and build those "zero" results (as mentioned above, these aren't
always 0 and depend on the aggregation function!).

The patch also adds two tests reproducing this issue in a fairly general
way (e.g., several aggregators, different aggregation functions) and
confirming the patch fixes the bug.

The test also includes two additional tests for COUNT aggregation, which
uncovered an incompatibility with Cassandra which is still not fixed -
so these tests are marked "xfail":

Refs #12477: Combining COUNT with GROUP by results with empty results
             in Cassandra, and one result with empty count in Scylla.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12715
2023-02-07 12:28:42 +02:00
Nadav Har'El
c00fcc80e5 test/cql-pytest: three tests for empty clustering keys
This patch adds three additional tests for empty (e.g., empty string)
clustering keys.

The first test disproves a worry that was raised in #12561 that perhaps
empty clustering keys only seem work, but they don't get written to
sstables. The new test verifies that there is no bug - they are written
and can be read correctly.

The second and third test reproduce issue #12749 - an empty clustering
should be allowed in a COMPACT STORAGE table only if there is a compound
(multi-column) clustering key. But as the tests demonstrate, 1. if there
is just one clustering column, Scylla gives the wrong error message, and
2. if there is a compound clustering key, Scylla doesn't allow an empty
key as it should.

As usual, all tests pass on Cassandra. The last two tests fail on
Scylla, so are marked xfail.

Refs #12561
Refs #12749

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12750
2023-02-07 12:28:41 +02:00
Nadav Har'El
cc207a9f44 Merge 'uda: improve checking whether UDFs are used in UDAs in DROP statements' from Wojciech Mitros
This patch fixes 2 issues with checking whether UDFs are used in UDAs:
1) UDFs types are not considered during the check, which prevents us from dropping a UDF that isn't used in any UDAs, but shares its name with one of them.
2) the REDUCEFUNC is not considered during the check, which allows dropping a UDF even though it's used in a UDA as the REDUCEFUNC.

Additionally, tests for these issues are added

Closes #12681

* github.com:scylladb/scylladb:
  udf: also check reducefunc to confirm that a UDF is not used in a UDA
  udf: fix dropping UDFs that share names with other UDFs used in UDAs
  pytest: add optional argument for new_function argument types
2023-02-06 19:07:26 +02:00
Wojciech Mitros
ef1dac813b udf: also check reducefunc to confirm that a UDF is not used in a UDA
When dropping a UDF we're checking if it's not begin used in any UDAs
and fail otherwise. However, we're only checking its state function
and final function, and it may also be used as its reduce function.
This patch adds the missing checks and a test for them.
2023-02-06 13:02:54 +01:00
Wojciech Mitros
49077dd144 udf: fix dropping UDFs that share names with other UDFs used in UDAs
Currently, when dropping a function, we only check if there exist
an aggregate that uses a function with the same name as its state
function or final function. This may cause the drop to fail even
when it's just another UDF with the same name that's used in the
aggregate, even when the actual dropped function is not used there.
This patch fixes this by checking whether not only the name of the
UDA's sfunc and finalfunc, but also their argument types.
2023-02-06 13:02:53 +01:00
Wojciech Mitros
8791b0faf5 pytest: add optional argument for new_function argument types
When multiple functions with the same name but different argument types
are created, the default drop statement for these functions will fail
because it does not include the argument types.
With this change, this problem can be worked around by specifying
argument types when creating the function, as this will cause the drop
statement to include them.
2023-02-06 13:02:19 +01:00
Avi Kivity
1e6cc9ca61 Merge 'storage_service: Enable Repair Based Node Operations (RBNO) by default for all node ops' from Asias He
Since 97bb2e47ff (storage_service: Enable Repair Based Node Operations (RBNO) by default for replace), RBNO was enabled by default for replace ops.

After more testing, we decided to enable repair based node operations by default for all node operations.

Closes #12173

* github.com:scylladb/scylladb:
  storage_service: Enable Repair Based Node Operations (RBNO) by default for all node ops
  test: Increase START_TIMEOUT
  test: Increase max-networking-io-control-blocks
  storage_service: Check node has left in node_ops_cmd::decommission_done
  repair: Use remote dc neighbors for everywhere strategy
2023-02-06 10:42:52 +02:00
Botond Dénes
c270c305c0 Merge 'Allow entire test suite to run with multiple compaction groups' from Raphael "Raph" Carvalho
New test/lib/scylla_test_case.hh, introduced in "tests: Add command line options for Scylla unit tests",
allows extension of the command line options provided by Seastar testing framework.
It allows all boost tests to process additional options without changing a single line of code.

Patch "test: Add x-log2-compaction-groups to Scylla test command line options" builds on that, allowing
all test cases to run with N compaction groups. Again, without changing a line of code in the tests.

Now all you have to do is:
./build/dev/test/boost/sstable_compaction_test -- --smp 1 --x-log2-compaction-groups 1
./test.py --mode=dev --x-log2-compaction-groups 1 --verbose

And it will run the test cases with as many groups as you wish.

./test.py passes successfully with parameter --x-log2-compaction-groups 1.

Closes #12369

* github.com:scylladb/scylladb:
  test.py: Add option to run scylla tests with multiple compaction groups
  test: Add x-log2-compaction-groups to Scylla test command line options
  test: Enable Scylla test command line options for boost tests
  tests: Add command line options for Scylla unit tests
  replica: table: Add debug log for number of compaction groups
  test: sstable_compaction_test: Fix indentation
  test: sstable_compaction_test: Make it work with compaction groups
  test: test_bloom_filter: Fix it with multiple compaction groups
  test: memtable_test: Fix it with multiple compaction groups
2023-02-03 06:35:15 +02:00
Asias He
6d7b4a896e test: Increase max-networking-io-control-blocks
The number is too low in the test and we saw

rpc: Connection is closed error

Inrease the number to the default 1000.
2023-02-02 11:11:22 +08:00
Raphael S. Carvalho
c25d8614a9 test: test_bloom_filter: Fix it with multiple compaction groups
With many compaction groups, the data:filter size ratio becomes small
with a small number of keys.

Test is adjusted to run another check with more keys if efficiency
is higher than expected, but not lower.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2023-02-01 20:14:51 -03:00