The manager will need system ks to get truncation record from, so add it
explicitly. Start-stop sequence no allows that
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The replayer code needs system keyspace to fetch truncation records
from, thus it needs this explicit dependency. By the time it runs system
keyspace is fully initialized already
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Implementation of task_manager's task that covers major keyspace compaction
on one shard.
Closes#12662
* github.com:scylladb/scylladb:
test: extend major keyspace compaction tasks test
compaction: create task manager's task for major keyspace compaction on one shard
- build: cmake: extract more subsystem out into its own CMakeLists.txt
- build: cmake: remove swagger_gen_files
- build: cmake: remove stale TODO comments
- build: cmake: expose scylla_gen_build_dir
- build: cmake: link against cryptopp
- build: cmake: add missing source to utils
- build: cmake: move lib sources into test-lib
- build: cmake: add test/perf
Closes#13059
* github.com:scylladb/scylladb:
build: cmake: add expr_test test
build: cmake: allow test to specify the sources
build: cmake: add test/perf
build: cmake: move lib sources into test-lib
build: cmake: add missing source to utils
build: cmake: link against cryptopp
build: cmake: expose scylla_gen_build_dir
build: cmake: remove stale TODO comments
build: cmake: remove swagger_gen_files
build: cmake: extract more subsystem out into its own CMakeLists.txt
This method requires callers to remember that the sstable is the collection of files on a filesystem and to know what exact directory they are all in. That's not going to work for object storage, instead, sstable should be moved between more abstract states.
This PR replaces move_to_new_dir() call with the change_state() one that accepts target sub-directory string and moves files around. Currently supported state changes:
* staging -> normal
* upload -> normal | staging
* any -> quarantine
All are pretty straightforward and move files between table basedir subdirectories with the exception that upload -> quarantine should move into upload/quarantine subdirectory. Another thing to keep in mind, that normal state doesn't have its subdir but maps directory to table's base directory.
Closes#12648
* github.com:scylladb/scylladb:
sstable: Remove explicit quarantization call
test: Move move_to_new_dir() method from sstable class
sstable, dist.-loader: Introduce and use pick_up_from_upload() method
sstables, code: Introduce and use change_state() call
distributed_loader: Let make_sstables_available choose target directory
some tests are compiled from more source files, so add an extra
parameter, so they can customize the sources.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
due to circular dependency: the .cc files under the root of
project references the symbols defined by the source files under
subdirectories, but the source files under subdirectories also
reference the symbols defined by the .cc files under the root
of project, the targets in test/perf do not compile. but
the general structure is created.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
This small series reorganizes the existing functional tests for aggregation (min, max, count) and adds additional tests for sum reproducing the strange (but Cassandra-compatible) behavior described in issue #13027.
Closes#13038
* github.com:scylladb/scylladb:
cql-pytest: add tests for sum() aggregate
test/cql-pytest: move aggregation tests to one file
The series fixes the `make_nonforwardable` reader, it shouldn't emit `partition_end` for previous partition after `next_partition()` and `fast_forward_to()`
Fixes: #12249Closes#12978
* github.com:scylladb/scylladb:
flat_mutation_reader_test: cleanup, seastar::async -> SEASTAR_THREAD_TEST_CASE
make_nonforwardable: test through run_mutation_source_tests
make_nonforwardable: next_partition and fast_forward_to when single_partition is true
make_forwardable: fix next_partition
flat_mutation_reader_v2: drop forward_buffer_to
nonforwardable reader: fix indentation
nonforwardable reader: refactor, extract reset_partition
nonforwardable reader: add more tests
nonforwardable reader: no partition_end after fast_forward_to()
nonforwardable reader: no partition_end after next_partition()
nonforwardable reader: no partition_end for empty reader
row_cache: pass partition_start though nonforwardable reader
- treewide: do not define/capture unused variables
- sstables/sstables: mark dummy variable for loop [[maybe_unused]]
- util/result_try: reference this explicitly
- raft: reference this explicitly
- idl-compiler: mark captured this used
- build: reenable unused-{variable,lambda-capture} warnings
Closes#12915
* github.com:scylladb/scylladb:
build: reenable unused-{variable,lambda-capture} warnings
test: reader_concurrency_semaphore_test: define target_memory in debug mode
api::failure_detector: mark set_phi_convict_threshold unimplemented
test: memtable_test: mark dummy variable for loop [[maybe_unused]]
idl-compiler: mark captured this used
raft: reference this explicitly
util/result_try: reference this explicitly
sstables/sstables: mark dummy variable for loop [[maybe_unused]]
treewide: do not define/capture unused variables
service: storage_service: clear _node_ops in batch
Since commit 73e258fc34, Scylla has partial
verification for the CLUSTERING ORDER BY clause in CREATE MATERIALIZED
VIEW. Specifically, invalid column names are rejected. But for reasons
explained in issue #12936 and in the test in this patch, Cassandra
demands that if CLUSTERING ORDER BY appears it must list all the
clustering columns, with no duplicates, and do so in the right order.
This patch replaces an existing test which suggested it is fine
(an extention over Cassandra) to accept a partial list of clustering
columns, by a test that verifies that such a partial list, or an
incorrectly-ordered list, or list with duplicates, should be rejected.
The new test fails on Scylla, and passes on Cassandra, so marked as xfail.
Refs #12936.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#12938
Before this patch, all scripts which use test/cql-pytest/run.py
looked for the Scylla executable as their first step. This is usually
the right thing to do, except in two cases where Scylla is *not* needed:
1. The script test/cql-pytest/run-cassandra.
2. The script test/alternator/run with the "--aws" option.
So in this patch we change run.py to only look for Scylla when actually
needed (the find_scylla() function is called). In both cases mentioned
above, find_scylla() will never get called and the script can work even
if Scylla was never built.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#13010
This flag designates that we should consume only one
partition from the underlying reader. This means that
attempts to move to another partition should cause an EOS.
When next_partition is called, the buffer could
contain partition_start and possibly static_row.
In this case clear_buffer_to_next_partition will
not remove anything from the buffer and the
reader position should not change. Before this patch,
however, we used to set _end_of_stream=false,
which violated the forwardable-reader
contract - the data of the next partition
was emitted after the data of the first partition
without intermediate EOS.
This bug was found when debugging
test_make_nonforwardable_from_mutations_as_mutation_source flakiness.
A corresponding focused test_make_forwardable_next_partition
has been added to exercise this problem.
This patch fixes the problem with method fast_forward_to
which is similar to the one with next_partition, no
partition_end should be injected for the partition if
fast_forward_to was called inside it.
Before the patch, nonforwardable reader injected
partition_end unconditionally. This caused problems
in case next_partition() was called, the downstream
reader might have already injected its own
partition_end marker, and the one from nonforwardable
reader was a duplicate.
Fixes: #12249
The patch introduces the _partition_is_open flag,
inject partition_end only if there was some data
in the input reader.
A simple unit test has been added for
the nonforwardable reader which checks this
new behaviour.
The WASM UDF implementation has changed since the last time the docs
were written. In particular, the Rust helper library has been
released, and using it should be the recommended method.
Some decisions that were only experimental at the start, were also
"set in stone", so we should refer to them as such.
The docs also contain some code examples. This patch adds tests for
these examples to make sure that they are not wrong and misleading.
Closes#12941
small_vector should be feature-wise compatible with std::vector<>,
let's add operator<=> for it.
also, there is not needd to define operator!=() explicitly, C++20
define this for us if operator==() is defined, so let's drop it.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#13032
without C++23 `std::ranges::repeat_view`, it'd be cumbersume to
implement a loop without dummy variable. this change helps to
silence following warning:
```
test/boost/memtable_test.cc:1135:26: error: unused variable 'value' [-Werror,-Wunused-variable]
for (int value : boost::irange<int>(0, num_flushes)) {
^
```
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
these warnings are found by Clang-17 after removing
`-Wno-unused-lambda-capture` and '-Wno-unused-variable' from
the list of disabled warnings in `configure.py`.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
This patch adds regression tests for the strange (but Cassandra-compatible)
behavior described in issue #13027 - that sum of no results returns 0
(not null or nothing), and if also asking for p, we get a null there too.
Refs #13027.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
We had separate test files test_minmax.py and test_count.py but the
separate was artificial (and test_count.py even had one test using
min()). Now I that want to add another test for sum(), I don't know
where to put it. So in this patch I combine test_minmax.py and
test_count.py into one test file - test_aggregate.py, and we can
later add sum() tests in the same file.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
test_broadcast_kv_store does not use await or yield at all, so
there is no need to mark it with "asyncio" mark.
tested using
```
SCYLLA_HOME=$HOME/scylla build/cmake/scylla --overprovisioned --developer-mode=yes --consistent-cluster-management=true --experimental-features=broadcast-tables
...
pytest broadcast_tables/test_broadcast_tables.py
```
the test still passes.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#13006
Recently, we overhauled the error handling of UNSET_VALUE in various
places where it is not allowed. This patch adds two more regression
tests for this error handling. Both tests pass on Scylla today, pass
on Cassandra, but fail on earlier Scylla (e.g., I tested 5.1.5):
The first test does INSERT into clustering key UNSET_VALUE.
An UNSET_VALUE is designed to skip part of the write - not an entire
write - so this attempt should fail - not silently be skipped.
The write indeed fails with an error on Cassandra, and on recent
Scylla, but silently did nothing in older Scylla which leads this
test to fail there.
The second test does the same thing with LWT (adding an "IF NOT EXISTS")
added to the insert. Scylla's failure here was even more spectacular -
it crashed (as reported in issue #13001) instead of silently skipping
the right. The test passes on Scylla today and on Cassandra, which
both report the failure cleanly.
Refs #13001.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#13007
Cassandra is very strict in the CLUSTERING ORDER BY clause which it
allows when creating a materialized view - if it appears, it must
list all the clustering columns of the view. Scylla is less strict -
a subset of the clustering columns may be specified. But Scylla was
*too* lenient - a user could specify non-clustering columns and even
non-existent columns and Scylla would not fail the MV creation.
This patch fixes that - with it MV creation fails if anything besides
clustering columns are listed on CLUSTERING ORDER BY.
An xfailing test we had for this case no longer fails after this
patch so its xfail mark is removed. We also add a few more corner
cases to the tests.
This patch also fixs one C++ test which had exactly the error that this
patch detects - the test author tried to use the partition key, instead
of the clustering key, in CLUSTERING ORDER BY (this error had no effect
because the specified order, "asc", was the default anyway).
Fixes#10767
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#12885
Said method can now throw `std::bad_alloc` since aab5954. All call-sites should have been adapted in the series introducing the throw, but some managed to slip through because the oom unit test didn't run in debug mode. This series fixes the remaining unpatched call-sites and makes sure the test runs in debug mode too, so leaks like this are detected.
Fixes: #12767Closes#12756
* github.com:scylladb/scylladb:
test/boost/reader_concurreny_semaphore_test: run oom protection tests in debug mode
treewide: adapt to throwing reader_concurrency_semaphore::consume()
since "token()" computes the token for a given partition key,
if we pass the key of the wrong type, it should reject.
in this change,
* we validate the keys before returning the "token()" function.
* drop the "xfail" decorator from two of the tests. they pass
now after this fix.
* change the tests which previously passed the wrong number of
arguments containing null to "token()" and expect it to return
null, so they verify that "token()" should reject these
arguments with the expected error message.
Fixes#10448
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#12991
Aggregation query on counter column is failing because forward_service is looking for function with counter as an argument and such function doesn't exist. Instead the long type should be used.
Fixes: #12939Closes#12963
* github.com:scylladb/scylladb:
test:boost: counter column parallelized aggregation test
service:forward_service: use long type when column is counter
It's known that reading large cells in reverse cause large allocations.
Source: https://github.com/scylladb/scylladb/issues/11642
The loading is preliminary work for splitting large partitions into
fragments composing a run and then be able to later read such a run
in an efficiency way using the position metadata.
The splitting is not turned on yet, anywhere. Therefore, we can
temporarily disable the loading, as a way to avoid regressions in
stable versions. Large allocations can cause stalls due to foreground
memory eviction kicking in.
The default values for position metadata say that first and last
position include all clustering rows, but they aren't used anywhere
other than by sstable_run to determine if a run is disjoint at
clustering level, but given that no splitting is done yet, it
does not really matter.
Unit tests relying on position metadata were adjusted to enable
the loading, such that they can still pass.
Fixes#11642.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Closes#12979
Task manager task implementation that covers the major keyspace
compaction which can be start through /storage_service/keyspace_compaction/
api.
Closes#12661
* github.com:scylladb/scylladb:
test: add test for major keyspace compaction tasks
compaction: create task manager's task for major keyspace compaction
compaction: copy run_on_existing_tables to task_manager_module.cc
compaction: add major_compaction_task_impl
compacition: add pure virtual compaction_task_impl
compaction: add compaction module getter to compaction manager
Some test cases can be made a bit more compact by using the sugar provided by the aforementioned sugar
Closes#12965
* github.com:scylladb/scylladb:
test: Make use of reusable_sst default format
tests: Use reusable_sst() where applicable
The sstable_test_env::reusable_sst() has default value for the format
argument. Patch the test cases that don't use one while at it.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The reusable_sst() is intented to be used to load the pre-existing
sstable from the test/resources directory and .load() them. Some test
cases, however, still do it "by hand".
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
it already includes the necessary bits used by test-perf, so let's
just link the latter to the former.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
BUILD_TESTING is an option exposed by CTest module, so let's
include CTest module, and check if BUILD_TESTING is enabled
before include boost based tests.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
The initial intent was to reduce the fanout of shared_sstable.hh through
v.u.g.hh -> cql_test_env.hh chain, but it also resulted in some shots
around v.u.g.hh -> database.hh inclusion.
By and large:
- v.u.g.hh doesn't need database.hh
- cql_test_env.hh doesn't need v.u.g.hh (and thus -- the
shared_sstable.hh) but needs database.hh instead
- few other .cc files need v.u.g.hh directly as they pulled it via
cql_test_env.hh before
- add forward declarations in few other places
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closes#12952
