It turns out that Cassandra handles a restriction like `(c2) = (1)` just
like `c2 = 1`, and is not limited like multi-column restrictions. In
particular, this query works despite missing "c1", and may also use an
index if c2 is indexed.
But currently in Scylla, `(c2) = (1)` is handled like a multi-column
restriction, so complains if c2 is not the first clustering key column,
and cannot use an index.
This patch adds several tests demonstrating this difference between
Scylla and Cassandra (#13250). The xfailing tests pass on Cassandra
but fail on Scylla.
Refs #13250
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#13252
To apply topology_change commands group0_state_machine needs to have an
access to the storage service to support topology changes over raft.
Message-Id: <20230316112801.1004602-10-gleb@scylladb.com>
Add a function that allows waiting for a state change of a raft server.
It is useful for a user that wants to know when a node becomes/stops
being a leader.
Message-Id: <20230316112801.1004602-4-gleb@scylladb.com>
The REST test test_storage_service.py::test_toppartitions_pk_needs_escaping
was flaky. It tests the toppartition request, which unfortunately needs
to choose a sampling duration in advance, and we chose 1 second which we
considered more than enough - and indeed typically even 1ms is enough!
but very rarely (only know of only one occurance, in issue #13223) one
second is not enough.
Instead of increasing this 1 second and making this test even slower,
this patch takes a retry approach: The tests starts with a 0.01 second
duration, and is then retried with increasing durations until it succeeds
or a 5-seconds duration is reached. This retry approach has two benefits:
1. It de-flakes the test (allowing a very slow test to take 5 seconds
instead of 1 seconds which wasn't enough), and 2. At the same time it
makes a successful test much faster (it used to always take a full
second, now it takes 0.07 seconds on a dev build on my laptop).
A *failed* test may, in some cases, take 10 seconds after this patch
(although in some other cases, an error will be caught immediately),
but I consider this acceptable - this test should pass, after all,
and a failure indicates a regression and taking 10 seconds will be
the last of our worries in that case.
Fixes#13223.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#13238
This series cleans up unit test in preparation for PR #12994.
Helpers are added (or reused) to not rely on specific sstable generation numbers where possible (other than loading reference sstables that are committed to the repo with given generation numbers), and to generate the sstables for tests easily, taking advantage of generation management in `sstable_test_env`, `table_for_tests`, or `replica::table` itself.
Closes#13242
* github.com:scylladb/scylladb:
test: add verify_mutation helpers.
test: add make_sstable_containing memtable
test: table_for_tests: add make_sstable function
test: sstable_test_env: add make_sst_factory methods
test: sstable_compaction_test: do not rely on specific generations
tests: use make_sstable defaults as much as possible
test: sstable_test_env: add make_table_for_tests
test: sstable_datafile_test: do not rely on sepecific sstable generations
test: sstable_test_env: add reusable_sst(shared_sstable)
sstable: expose get_storage function
test: mutation_reader_test: create_sstable: do not rely on specific generations
test: mutation_reader_test: do_test_clustering_order_merger_sstable_set: rely on test_envsstable generation
test: mutation_reader_test: combined_mutation_reader_test: define a local sst_factory function
test: mutation_reader_test: do not use tmpdir
test: use big format by default
test: sstable_compaction_test: use highest sstable version by default
test: test_env: make_db_config: set cfg host_id
test: sstable_datafile_test: fixup indentation
test: sstable_datafile_test: various tests: do_with_async
test: sstable_3_x_test: validate_read, sstable_assertions: get shared_sstable
test: sstable_3_x_test: compare_sstables: get shared_sstable
test: sstable_3_x_test: write_sstables: return shared_sstable
test: sstable_3_x_test: write, compare, validate_sstables: use env.tempdir
test: sstable_3_x_test: compacted_sstable_reader: do not reopen compacted_sst
test: lib: test_services: delete now unused stop_and_keep_alive
test: sstable_compaction_test: use deferred_stop to stop table_for_tests
test: sstable_compaction_test: compound_sstable_set_incremental_selector_test: do_with_async
test: sstable_compaction_test: sstable_needs_cleanup_test: do_with_async
test: sstable_compaction_test: leveled_05: fixup indentation
test: sstable_compaction_test: leveled_05: do_with_async
test: sstable_compaction_test: compact_02: do_with_async
test: sstable_compaction_test: compact_sstables: simplify variable allocation
test: sstable_compaction_test: compact_sstables: reindent
test: sstable_compaction_test: compact_sstables: use thread
test: sstable_compaction_test: sstable_rewrite: simplify variable allocation
test: sstable_compaction_test: sstable_rewrite: fixup indentation
test: sstable_compaction_test: sstable_rewrite: do_with_async
test: sstable_compaction_test: compact: fixup indentation
test: sstable_compaction_test: compact: complete conversion to async thread
test: sstable_compaction_test: compaction_manager_basic_test: rename generations to idx
This patch addes a reproducing test for issue #13241, where attempting
a SELECT restriction (b,c,d) IN ((1,2)) - where the tuple is shorter
than needed - crashes Scylla (on segmentation fault) instead of
generating a clean error as it should (and as done on Cassandra).
The test also demonstractes that if the tuple is longer than needed
(instead of shorter), the behavior is correct, and it is also
correct if "=" is used instead of IN. Only the combination of IN
and too-short tuple seems to be broken - but broken in a bad way
(can be used to crash Scylla).
Because the test crashes Scylla when fails, it is marked "skip".
Refs #13241
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#13244
this is the 14rd changeset of a series which tries to give an overhaul to the CMake building system. this series has two goals:
- to enable developer to use CMake for building scylla. so they can use tools (CLion for instance) with CMake integration for better developer experience
- to enable us to tweak the dependencies in a simpler way. a well-defined cross module / subsystem dependency is a prerequisite for building this project with the C++20 modules.
this changeset includes following changes:
- build: cmake: promote add_scylla_test() to test/
- build: cmake: add all tests
Closes#13220
* github.com:scylladb/scylladb:
build: cmake: add all tests
build: cmake: promote add_scylla_test() to test/
Some unit tests want to change the sstable::_dir on the fly. However,
the sstable::_dir is going away, so it needs a yet another virtual call
on storage driver.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closes#13213
table_for_tests uses a sstables manager to generate sstables
and gets the new generation from
table.calculate_generation_for_new_table().
The version to use is either the highest supported or
an ad-hoc version passed to make_sstable.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
The tests extensively use a `std::function<shared_sstable()>`
to generate new tables.
Rather than handcrafting them all over the place,
let sstable_test_env return such factory given a schema
(and another entry point that also gets a version)
and that uses the embedded generation_factory in the test_env
to generate new sstables with unique generations.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
No need to maintain a static generation numbers in the test.
Let the sstable_test_env dispatch sstable generations automatically
And use the generated sstable themselves for reference rather
than their generation numbers.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Add a few goodies to sstable_test_env to extend
entry points with default params for make_sstable
and reusable_sst.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Wrap table_for_tests ctor to pass the env sstables_manager
as well as the temporary directory path, as this is the
most common use case, and in preparation for adding
a make_sstable method in table_for_tests.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
There is no need to use specific generations in the test, just
rely on the ones sstable_test_env generates.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Allow generating a sstable object from an existing
sstable to get the directory, generation, and version
from it, rather than passing them to reusable_sst
from other sources - since the intention is
to get a new sstable object based on an existing
sstable that was generated by the test.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than maintaining a running generation number,
use the default env.make_sstable(s) in sst_factory
and collect the expected generations from the resulting
shared sstable.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
For generating shared_sstables with increasing generations
(using the test_env make_sstable generations) and a given level.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
No need to pass the big format explicitly as it's
set by default by make_sstable and it is never overriden.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Tests should just generate the highest sstable version
available. There is no need to ontinue testing old versions,
in particular partially supported ones like "la".
Use also the default values for sstable::format_types, buffer_size,
etc. if there's no particular need to override them.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Pass the test-generated shared_sstable to validate_read
and then to sstable_assertions so it can be used
for make_sstable version and generation params.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Just use the one we created during compaction
for verification so we won't have to rely on a particular
generation/version.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than calling cf.stop_and_keep_alive() before the test exits.
since it must be stopped also on failure.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
We already use test_env::do_with_async in this function
but we didn't take full advantage of it to simplify the
implementation.
Do that before further changes are made.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
generations to idx
The function used `calculate_generation_for_new_table` for
the sstables generation. The so-called `generations` are just used
to generate key indices.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
The `storage_options` describes where sstables should be located. Currently the object reside on keyspace_metadata, but is thus not available at the place it's needed the most -- the `table::make_sstable()` call. This set converts keyspace_metadata::storage_opts to be lw-shared-ptr and shares the ptr with class table.
refs: #12523 (detached small change from large PR)
Closes#13212
* github.com:scylladb/scylladb:
table: Keep storage options lw-shared-ptr
keyspace_metadata: Make storage options lw-shared-ptr
this change should address the FTBFS with Clang-17.
turns out we are comparing a mutation with an
optimized_optional<mutation>. and Clang-17 does not want to convert the
LHS, which is a mutation to optimized_optional<mutation> for performing
the comparison using operator==(const optimized_optional<mutation>&),
desipte that optimized_optional(const T& obj) is not marked explicit.
this is understandable.
so, in this change, instead of relying on the implicit conversion, we
just
* check if the optional actually holds a value
* and compare the value by deferencing the optional.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#13196
this series intends to deprecate `::join()`, as it always materializes a range into a concrete string. but what we always want is to print the elements in the given range to stream, or to a seastar logger, which is backed by fmtlib. also, because fmtlib offers exactly the same set of features implemented by to_string.hh, this change would allow us to use fmtlib to replace to_string.hh for better maintainability, and potentially better performance. as fmtlib is lazy evaluated, and claims to be performant under most circumstances.
Closes#13163
* github.com:scylladb/scylladb:
utils: to_string: move join to namespace utils
treewide: use fmt::join() when appropriate
row_cache: pass "const cache_entry" to operator<<
