Commit Graph

2199 Commits

Author SHA1 Message Date
Nadav Har'El
73e258fc34 materialized views: verify CLUSTERING ORDER BY clause
Cassandra is very strict in the CLUSTERING ORDER BY clause which it
allows when creating a materialized view - if it appears, it must
list all the clustering columns of the view. Scylla is less strict -
a subset of the clustering columns may be specified. But Scylla was
*too* lenient - a user could specify non-clustering columns and even
non-existent columns and Scylla would not fail the MV creation.
This patch fixes that - with it MV creation fails if anything besides
clustering columns are listed on CLUSTERING ORDER BY.

An xfailing test we had for this case no longer fails after this
patch so its xfail mark is removed. We also add a few more corner
cases to the tests.

This patch also fixs one C++ test which had exactly the error that this
patch detects - the test author tried to use the partition key, instead
of the clustering key, in CLUSTERING ORDER BY (this error had no effect
because the specified order, "asc", was the default anyway).

Fixes #10767

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes #12885
2023-02-27 15:09:42 +02:00
Avi Kivity
6f88dc8009 Merge 'Fix memory leaks caused by throwing reader_concurrency_semaphore::consume()' from Botond Dénes
Said method can now throw `std::bad_alloc` since aab5954. All call-sites should have been adapted in the series introducing the throw, but some managed to slip through because the oom unit test didn't run in debug mode. This series fixes the remaining unpatched call-sites and makes sure the test runs in debug mode too, so leaks like this are detected.

Fixes: #12767

Closes #12756

* github.com:scylladb/scylladb:
  test/boost/reader_concurreny_semaphore_test: run oom protection tests in debug mode
  treewide: adapt to throwing reader_concurrency_semaphore::consume()
2023-02-27 12:27:30 +02:00
Botond Dénes
61e67b865a Merge 'service:forward_service: use long type instead of counter in function mocking' from Michał Jadwiszczak
Aggregation query on counter column is failing because forward_service is looking for function with counter as an argument and such function doesn't exist. Instead the long type should be used.

Fixes: #12939

Closes #12963

* github.com:scylladb/scylladb:
  test:boost: counter column parallelized aggregation test
  service:forward_service: use long type when column is counter
2023-02-24 15:25:10 +02:00
Raphael S. Carvalho
d73ffe7220 sstables: Temporarily disable loading of first and last position metadata
It's known that reading large cells in reverse cause large allocations.
Source: https://github.com/scylladb/scylladb/issues/11642

The loading is preliminary work for splitting large partitions into
fragments composing a run and then be able to later read such a run
in an efficiency way using the position metadata.

The splitting is not turned on yet, anywhere. Therefore, we can
temporarily disable the loading, as a way to avoid regressions in
stable versions. Large allocations can cause stalls due to foreground
memory eviction kicking in.
The default values for position metadata say that first and last
position include all clustering rows, but they aren't used anywhere
other than by sstable_run to determine if a run is disjoint at
clustering level, but given that no splitting is done yet, it
does not really matter.

Unit tests relying on position metadata were adjusted to enable
the loading, such that they can still pass.

Fixes #11642.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>

Closes #12979
2023-02-24 12:14:18 +02:00
Michał Jadwiszczak
4c6675bf1a test:boost: counter column parallelized aggregation test 2023-02-24 10:24:23 +01:00
Botond Dénes
624d176b3b Merge 'Refine usage of sstable_test_env::reusable_sst() method' from Pavel Emelyanov
Some test cases can be made a bit more compact by using the sugar provided by the aforementioned sugar

Closes #12965

* github.com:scylladb/scylladb:
  test: Make use of reusable_sst default format
  tests: Use reusable_sst() where applicable
2023-02-23 12:50:06 +01:00
Botond Dénes
a5979c0662 Merge 'treewide: remove invalid defaulted move ctor' from Kefu Chai
- test/boost/chunked_vector_test: remove defaulted exception_safe_class's move ctor
- tools/scylla-sstable: remove defaulted move ctor
- sstables/mx/partition_reversing_data_source: remove defaulted move ctor
- cql3/statements/truncate_statement: remove defaulted move ctor

Closes #12914

* github.com:scylladb/scylladb:
  test/boost/chunked_vector_test: remove defaulted exception_safe_class's move ctor
  tools/scylla-sstable: remove defaulted move ctor
  sstables/mx/partition_reversing_data_source: remove defaulted move ctor
  cql3/statements/truncate_statement: remove defaulted move ctor
2023-02-23 12:50:05 +01:00
Pavel Emelyanov
5b311bb724 test: Make use of reusable_sst default format
The sstable_test_env::reusable_sst() has default value for the format
argument. Patch the test cases that don't use one while at it.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-02-22 17:04:10 +03:00
Pavel Emelyanov
7aabffff19 tests: Use reusable_sst() where applicable
The reusable_sst() is intented to be used to load the pre-existing
sstable from the test/resources directory and .load() them. Some test
cases, however, still do it "by hand".

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-02-22 17:03:15 +03:00
Pavel Emelyanov
f51762c72a headers: Refine view_update_generator.hh and around
The initial intent was to reduce the fanout of shared_sstable.hh through
v.u.g.hh -> cql_test_env.hh chain, but it also resulted in some shots
around v.u.g.hh -> database.hh inclusion.

By and large:
- v.u.g.hh doesn't need database.hh
- cql_test_env.hh doesn't need v.u.g.hh (and thus -- the
  shared_sstable.hh) but needs database.hh instead
- few other .cc files need v.u.g.hh directly as they pulled it via
  cql_test_env.hh before
- add forward declarations in few other places

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>

Closes #12952
2023-02-22 09:32:30 +02:00
Calle Wilund
64102780fe commitlog: Use static (reused) regex for (left over) descriptor parse
Refs #11710

Allows reusing regex for segment matching (for opening left-over segments after crash).
Should remove any stalls caused by commitlog replay preparation.

v2: Add unit test for descriptor parsing

Closes #12112
2023-02-21 18:34:04 +02:00
Tomasz Grabiec
c8e2bf1596 db: schema_tables: Optimize schema merge
Currently, applying a schema change on a replica works like this:

  Collect all affected keyspaces from incoming mutations
  Read current state of schema
  Apply the mutations
  Read new state of schema

The "Read ... state of schema" step reads all kinds of schema
objects. In particular, to read the "table" objects, it does the
following:

  for every affected keyspace k:
       read all mutations from system_schema.tables for k
       extract all existing table names from those mutations
       for every existing table:
             read mutations from {tables, columns, indexes, view_virtual_columns, ...} for that table

As you can see, the number of reads performed is O(nr tables in a
keyspace), not O(nr tables in a change). This means that making a
sequence of schema changes, like adding a table, is quadratic.

Another aspect which magnifies this is that we don't read those tables
using a single scan, but issue individual queries for each table
separately.

This patch optimizes this by considering only affected tables when
reading schema for the purpose of diff calculation.

When mutations contain multi-table deletions, we still read the
set of tables, like before. This could be optimized by looking
at the database to get the list, but it's not part of the patch.

I tested this using a test case provided by Kamil (kbr-scylla@53fe154)

  ./test.py --mode debug test_many_schema_changes -s

The test bootstraps a cluster and then creates about 40 schema
changes. Then a new node is bootstrapped and replays those changes via
group0.

In debug mode, each change takes roughly 2s to process before the
patch, and 0.5s after the patch.

The whole replay is reduced to 56% of what was before:

Before (1m19s) :

INFO  2023-01-20 19:44:35,848 [shard 0] raft_group0 - setup_group0: ensuring that the cluster has fully upgraded to use Raft...
INFO  2023-01-20 19:45:54,844 [shard 0] raft_group0 - setup_group0: waiting for peers to synchronize state...

After (45s):

INFO  2023-01-20 22:02:51,869 [shard 0] raft_group0 - setup_group0: ensuring that the cluster has fully upgraded to use Raft...
INFO  2023-01-20 22:03:36,834 [shard 0] raft_group0 - setup_group0: waiting for peers to synchronize state...

Closes #12592

Closes #12592
2023-02-21 17:26:57 +02:00
Botond Dénes
3c30531202 Merge 'test: mutation_test: Fix sporadic failure due to continuity mismatch' from Tomasz Grabiec
In test_v2_apply_monotonically_is_monotonic_on_alloc_failures we
generate mutations with non-full continuity, so we should pass
is_evictable::yes to apply_monotonically(). Otherwise, it will assume
fully-continuous versions and not try to maintain continuity by
inserting sentinels.

This manifested in sporadic failures on continuity check.

Fixes #12882

Closes #12921

* github.com:scylladb/scylladb:
  test: mutation_test: Fix sporadic failure due to continuity mismatch
  test: mutation_test: Fix copy-paste mistake in trace-level logging
2023-02-20 12:46:15 +01:00
Kefu Chai
df63e2ba27 types: move types.{cc,hh} into types
they are part of the CQL type system, and are "closer" to types.
let's move them into "types" directory.

the building systems are updated accordingly.

the source files referencing `types.hh` were updated using following
command:

```
find . -name "*.{cc,hh}" -exec sed -i 's/\"types.hh\"/\"types\/types.hh\"/' {} +
```

the source files under sstables include "types.hh", which is
indeed the one located under "sstables", so include "sstables/types.hh"
instea, so it's more explicit.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes #12926
2023-02-19 21:05:45 +02:00
Kefu Chai
ee97c332d9 test/boost/chunked_vector_test: remove defaulted exception_safe_class's move ctor
because it has a member variable whose type is a reference. and a
reference cannot be reassigned. this silences following warning from Clang:

```
/home/kefu/dev/scylladb/test/boost/chunked_vector_test.cc:152:27: error: explicitly defaulted move assignment operator is implicitly deleted [-Werror,-Wdefaulted-function-deleted]
    exception_safe_class& operator=(exception_safe_class&&) = default;
                          ^
/home/kefu/dev/scylladb/test/boost/chunked_vector_test.cc:132:31: note: move assignment operator of 'exception_safe_class' is implicitly deleted because field '_esc' is of reference type 'exception_safety_checker &'
    exception_safety_checker& _esc;
                              ^
```

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-02-19 12:58:22 +08:00
Michał Chojnowski
e88f590eda sstables: partition_index_cache: clean up an unused type alias
`list_ptr` is a type alias that isn't used in any meaningful way. Remove it.

Closes #10978
2023-02-17 17:58:26 +03:00
Tomasz Grabiec
2ae8f74cec test: mutation_test: Fix sporadic failure due to continuity mismatch
In test_v2_apply_monotonically_is_monotonic_on_alloc_failures we
generate mutations with non-full continuity, so we should pass
is_evictable::yes to apply_monotonically(). Otherwise, it will assume
fully-continuous versions and not try to maintain continuity by
inserting sentinels.

This manifested in sporadic failures on continuity check.

Fixes #12882
2023-02-17 14:43:32 +01:00
Tomasz Grabiec
22063713d7 test: mutation_test: Fix copy-paste mistake in trace-level logging 2023-02-17 14:42:47 +01:00
Kefu Chai
05ecc3f1c9 build: cmake: add unit tests
this change is based on Botond Dénes's change which gave an overhaul
to the original CMake building system. this change is not enough
to build tests with CMake, as we still need to sort out the
dependencies.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-02-17 18:41:40 +08:00
Botond Dénes
0961a3f79b test/boost/reader_concurreny_semaphore_test: run oom protection tests in debug mode
Said tests require on being run with a limited amount of memory to be
really useful. When the memory amount is unexpected, they silently exit.
Which is exactly what they did in debug mode too, where the amount of
memory available cannot be controlled.
Disable the check in debug mode.
2023-02-17 00:46:56 -05:00
Avi Kivity
e2f6e0b848 utils: move hashing related files to utils/ module
Closes #12884
2023-02-17 07:19:52 +02:00
Botond Dénes
79bf347e04 Merge 'Remove sstables::test_setup in favor of sstables::test_env' from Pavel Emelyanov
The former is a convenience wrapper over the latter. There's no real benefit in using it, but having two test_env-s is worse than just one.

Closes #12794

* github.com:scylladb/scylladb:
  sstable_utils: Move the test_setup to perf/
  sstable_utils: Remove unused wrappers over test_env
  sstable_test: Open-code do_with_cloned_tmp_directory
  sstable_test: Asynchronize statistics_rewrite case
  tests: Replace test_setup::do_with_tmp_directory with test_env::do_with(_async)?
2023-02-17 07:09:34 +02:00
Avi Kivity
69a385fd9d Introduce schema/ module
Schema related files are moved there. This excludes schema files that
also interact with mutations, because the mutation module depends on
the schema. Those files will have to go into a separate module.

Closes #12858
2023-02-15 11:01:50 +02:00
Avi Kivity
c5e4bf51bd Introduce mutation/ module
Move mutation-related files to a new mutation/ directory. The names
are kept in the global namespace to reduce churn; the names are
unambiguous in any case.

mutation_reader remains in the readers/ module.

mutation_partition_v2.cc was missing from CMakeLists.txt; it's added in this
patch.

This is a step forward towards librarization or modularization of the
source base.

Closes #12788
2023-02-14 11:19:03 +02:00
Raphael S. Carvalho
d6fe99abc4 replica: table: Update stats for newly added SSTables
Patch 55a8421e3d fixed an inefficiency when rebuilding
statistics with many compaction groups, but it incorrectly removed
the update for newly added SSTables. This patch restores it.
When a new SSTable is added to any of the groups, the stats are
incrementally updated (as before). On compaction completion,
statistics are still rebuilt by simply iterating through each
group, which keeps track of its own stats.
Unit tests are added to guarantee the stats are correct both after
compaction completion and memtable flush.

Fixes #12808.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>

Closes #12834
2023-02-14 10:28:53 +02:00
Nadav Har'El
efed973dd3 Merge 'cql3: convert LWT IF clause to expressions' from Avi Kivity
LWT `IF` (column_condition) duplicates the expression prepare and evaluation code. Annoyingly,
LWT IF semantics are a little different than the rest of CQL: a NULL equals NULL, whereas usually
NULL = NULL evaluates to NULL.

This series converts `IF` prepare and evaluate to use the standard expression code. We employ
expression rewriting to adjust for the slightly different semantics.

In a few places, we adjust LWT semantics to harmonize them with the rest of CQL. These are pointed
out in their own separate patches so the changes don't get lost in the flood.

Closes #12356

* github.com:scylladb/scylladb:
  cql3: lwt: move IF clause expression construction to grammar
  cql3: column_condition: evaluate column_condition as a single expression
  cql3: lwt: allow negative list indexes in IF clause
  cql3: lwt: do not short-circuit col[NULL] in IF clause
  cql3: column_condition: convert _column to an expression
  cql3: expr: generalize evaluation of subscript expressions
  cql3: expr: introduce adjust_for_collection_as_maps()
  cql3: update_parameters: use evaluation_inputs compatible row prefetch
  cql3: expr: protect extract_column_value() from partial clustering keys
  cql3: expr: extract extract_column_value() from evaluation machinery
  cql3: selection: introduce selection_from_partition_slice
  cql3: expr: move check for ordering on duration types from restrictions to prepare
  cql3: expr: remove restrictions oper_is_slice() in favor of expr::is_slice()
  cql3: column_condition: optimize LIKE with constant pattern after preparing
  cql3: expr: add optimizer for LIKE with constant pattern
  test: lib: add helper to evaluate an expression with bind variables but no table
  cql3: column_condition: make the left-hand-side part of column_condition::raw
  cql3: lwt: relax constraints on map subscripts and LIKE patterns
  cql3: expr: fix search_and_replace() for subscripts
  cql3: expr: fix function evaluation with NULL inputs
  cql3: expr: add LWT IF clause variants of binary operators
  cql3: expr: change evaluate_binop_sides to return more NULL information
2023-02-13 16:30:24 +02:00
Pavel Emelyanov
fa5f5a3299 sstable_test_env: Remove working_sst helper
It's only used by the single test and apparently exists since the times
seastar was missing the future::discard_result() sugar

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>

Closes #12803
2023-02-13 16:30:24 +02:00
Avi Kivity
87c0d09d03 cql3: lwt: move IF clause expression construction to grammar
Instead of the grammar passing expression bits to column_condition,
have the grammar construct an unprepared expression and pass it as
a whole. column_condition::raw then uses prepare_expression() to
prepare it.

The call to validate_operation_on_durations() is eliminated, since it's
already done be prepare_expression().

Some tests adjusted for slightly different wording.
2023-02-12 17:28:36 +02:00
Avi Kivity
31ee13c0c9 cql3: expr: move check for ordering on duration types from restrictions to prepare
Both LWT IF clause and SELECT WHERE clause check that a duration type
isn't used in an ordered comparison, since duration types are unordered
(is 1mo more or less than 30d?). As a first step towards centralizing this
check, move the check from restrictions into prepare. When LWT starts using
prepare, the duplication will be removed.

The error message was changed: the word "slice" is an internal term, and
a comparison does not necessarily have to be in a restriction (which is
also an internal term).

Tests were adjusted.
2023-02-12 17:17:01 +02:00
Avi Kivity
db2fa44a9a cql3: expr: add optimizer for LIKE with constant pattern
Compiling a pattern is expensive and so we should try to do it
at prepare time, if the pattern is a constant. Add an optimizer
that looks for such cases and replaces them with a unary function
that embeds the compiled pattern.

This isn't integrated yet with prepare_expr(), since the filtering
code isn't ready for generic expressions. Its first user will be LWT,
which contains the optimization already (filtering had it as well,
but lost it sometime during the expression rewrite).

A unit test is added.
2023-02-12 17:16:58 +02:00
Avi Kivity
ecdd49317a cql3: expr: add LWT IF clause variants of binary operators
LWT IF clause interprets equality differently from SQL (and the
rest of CQL): it thinks NULL equals NULL. Currently, it implements
binary operators all by itself so the fact that oper_t::EQ (and
friends) means something else in the rest of the code doesn't
bother it. However, we can't unify the code (in
column_condition.cc) with the rest of expression evaluation if
the meaning changes in different places.

To prepare for this, introduce a null_handling_style field to
binary_operator that defaults to `sql` but can be changed to
`lwt_nulls` to indicate this special semantic.

A few unit tests are added. LWT itself still isn't modified.
2023-02-12 17:03:03 +02:00
Botond Dénes
423df263f5 Merge 'Sanitize with_sstable_directory() helper in tests' from Pavel Emelyanov
The helping wrapper facilitates the usage of sharded<sstable_directory> for several test cases and the helper and its callers had deserved some cleanup over time.

Closes #12791

* github.com:scylladb/scylladb:
  sstable_directory_test: Reindent and de-multiline
  sstable_directory_test: Enlighten and rename sstable_from_existing_file
  sstable_directory_test: Remove constant parallelizm parameter
2023-02-10 07:11:38 +02:00
Tomasz Grabiec
402d5fd7e3 cache: Fix empty partition entries being left in cache in some cases
Merging rows from different partition versions should preserve the LRU link of
the entry from the newer version. We need this in case we're merging two last
dummy entries where the older dummy is already unlinked from the LRU. The
newer dummy could be the last entry which is still holding the partition
entry linked in the LRU.

The mutation_partition_v2 merging didn't take the LRU link from the newer
entry, and we could end up with the partition entry not having any entries
linked in the LRU.

Introduced in f73e2c992f.

Fixes #12778

Closes #12785
2023-02-09 23:03:23 +02:00
Pavel Emelyanov
47022bf750 sstable_test: Open-code do_with_cloned_tmp_directory
The statistics_rewrite case uses the helper that creates a copy of the
provided static directory, but it's the only user of this helper. It's
better to open-code it into the test case.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-02-09 17:17:48 +03:00
Pavel Emelyanov
19c1afb20a sstable_test: Asynchronize statistics_rewrite case
It is ran inside async context and can be coded in a shorter form
without using deeply nested then-s

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-02-09 17:17:23 +03:00
Pavel Emelyanov
85b8bae035 tests: Replace test_setup::do_with_tmp_directory with test_env::do_with(_async)?
The former helper is just a wrapper over the _async version of the
latter and also creates a tempdir and calls the fn with tempdir as an
argument. The test_env already has its own temp dir on board, so callers
can can be switched to using it.

Some test cases use the do_with_tmp_directory but generate chain of
futures without in fact using the async context. This patch addresses
that, so the change is not 100% mechanical unfortunately.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-02-09 17:11:31 +03:00
Pavel Emelyanov
f0212c7b68 sstable_directory_test: Reindent and de-multiline
Many tests using sstable directory wrapper have broken indentation with
previous patching. Fix it. No functional changes.

Also, while at it, convert multiline wrapper calls into one-line, after
previous patch these are short enough for that.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-02-09 16:00:53 +03:00
Pavel Emelyanov
ec02b0f706 sstable_directory_test: Enlighten and rename sstable_from_existing_file
It used to be the sstable maker for sstable::test_env / cql_test_env,
now sstables for tests are made via sstables manager explicitly, so the
guy can be remaned to something more relevant to its current status.

Also, de-mark its constructors as explicit to make callers look shorter.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-02-09 15:59:23 +03:00
Pavel Emelyanov
c843f7937b sstable_directory_test: Remove constant parallelizm parameter
It's 1 (one) all the time, just hard-code it internally

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-02-09 15:59:01 +03:00
Botond Dénes
b62d84fdba Merge 'Keep reshape and reshard logic in distributed loader' from Pavel Emelyanov
Now it's scattered between dist. loader and sstable directory code making the latter quite bloated. Keeping everything in distributed loader makes the sstable_directory code compact and easier to patch to support object storage backend.

Closes #12771

* github.com:scylladb/scylladb:
  sstable_directory: Rename remove_input_sstables_from_reshaping()
  sstable_directory: Make use of remove_sstables() helper
  sstable_directory: Merge output sstables collecting methods
  distributed_loader: Remove max_compaction_threshold argument from reshard()
  distributed_loader: Remove compaction_manager& argument from reshard()
  sstable_directory: Move the .reshard() to distributed_loader
  sstable_directory: Add helper to load foreign sstable
  sstable_directory: Add io-prio argument to .reshard()
  sstable_directory: Move reshard() to distributed_loader.cc
  distributed_loader: Remove compaction_manager& argument from reshape()
  sstable_directory: Move the .reshape() to distributed loader
  sstable_directory: Add helper to retrive local sstables
  sstable_directory: Add io-prio argument to .reshape()
  sstable_directory: Move reshape() to distributed_loader.cc
2023-02-09 10:01:44 +02:00
Avi Kivity
0f15ff740d cql3: expr: simplify user/debug formatting
We have a cql3::expr::expression::printer wrapper that annotates
an expression with a debug_mode boolean prior to formatting. The
fmt library, however, provides a much simpler alterantive: a custom
format specifier. With this, we can write format("{:user}", expr) for
user-oriented prints, or format("{:debug}", expr) for debug-oriented
prints (if nothing is specified, the default remains debug).

This is done by implementing fmt::formatter::parse() for the
expression type, can using expression::printer internally.

Since sometimes we pass expression element types rather than
the expression variant, we also provide a custom formatter for all
ExpressionElement Types.

Uses for expression::printer are updated to use the nicer syntax. In
one place we eliminate a temporary that is no longer needed since
ExpressionElement:s can be formatted directly.

Closes #12702
2023-02-08 12:24:58 +02:00
Pavel Emelyanov
e6e65c87d5 sstable_directory: Add io-prio argument to .reshard()
Now it gets one from this-> but the method is becoming static one in
distributed_loader which only has it as an argument. That's not big deal
as the current IO class is going to be derived from current sched group,
so this extra arg will go away at all some day.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-02-07 19:31:41 +03:00
Petr Gusev
95bf8eebe0 query_ranges_to_vnodes_generator: fix for exclusive boundaries
Let the initial range passed to query_partition_key_range
be [1, 2) where 2 is the successor of 1 in terms
of ring_position order and 1 is equal to vnode.
Then query_ranges_to_vnodes_generator() -> [[1, 1], (1, 2)],
so we get an empty range (1,2) and subsequently will
make a data request with this empty range in
storage_proxy::query_partition_key_range_concurrent,
which will be redundant.

The patch adds a check for this condition after
making a split in the main loop in process_one_range.

The patch does not attempt to handle cases where the
original ranges were empty, since this check is the
responsibility of the caller. We only take care
not to add empty ranges to the result as an
unintentional artifact of the algorithm in
query_ranges_to_vnodes_generator.

A test case is added in test_get_restricted_ranges.
The helper lambda check is changed so that not to limit
the number of ranges to the length of expected
ranges, otherwise this check passes without
the change in process_one_range.

Fixes: #12566

Closes #12755
2023-02-07 16:02:31 +02:00
Tomasz Grabiec
ccc8e47db1 Merge 'test/lib: introduce key_utils.hh' from Botond Dénes
We currently have two method families to generate partition keys:
* make_keys() in test/lib/simple_schema.hh
* token_generation_for_shard() in test/lib/sstable_utils.hh

Both work only for schemas with a single partition key column of `text` type and both generate keys of fixed size.
This is very restrictive and simplistic. Tests, which wanted anything more complicated than that had to rely on open-coded key generation.
Also, many tests started to rely on the simplistic nature of these keys, in particular two tests started failing because the new key generation method generated keys of varying size:
* sstable_compaction_test.sstable_run_based_compaction_test
* sstable_mutation_test.test_key_count_estimation

These two tests seems to depend on generated keys all being of the same size. This makes some sense in the case of the key count estimation test, but makes no sense at all to me in the case of the sstable run test.

Closes #12657

* github.com:scylladb/scylladb:
  test/lib/sstable_utils: remove now unused token_generation_for_shard() and friends
  test/lib/simple_schema: remove now unused make_keys() and friends
  test: migrate to tests::generate_partition_key[s]()
  test/lib/test_services: add table_for_tests::make_default_schema()
  test/lib: add key_utils.hh
  test/lib/random_schema.hh: value_generator: add min_size_in_bytes
2023-02-06 18:11:32 +01:00
Kamil Braun
56c4d246ef Merge 'Introduce recent_entries_map datatype to track least recent visited entries.' from Andrii Patsula
Fixes: https://github.com/scylladb/scylladb/issues/12309

Closes #12720

* github.com:scylladb/scylladb:
  service/raft: raft_group_registry: use recent_entries_map to store rate_limits in pinger. Fixes #12309
  utils: introduce recent_entries_map datatype to track least recent visited entries.
2023-02-06 18:01:26 +01:00
Botond Dénes
8efa9b0904 Merge 'Avoid qctx from view-builder methods of system_keyspace' from Pavel Emelyanov
The system_keyspace defines several auxiliary methods to help view_builder update system.scylla_views_builds_in_progress and system.built_views tables. All use global qctx thing.

It only takes adding view_builder -> system_keyspace dependency in order to de-static all those helpers and let them use query-processor from it, not the qctx.

Closes #12728

* github.com:scylladb/scylladb:
  system_keysace: De-static calls that update view-building tables
  storage_service: Coroutinize mark_existing_views_as_built()
  api: Unset column_famliy endpoints
  api: Carry sharded<db::system_keyspace> reference over
  view_builder: Add system_keyspace dependency
2023-02-06 12:44:40 +02:00
Botond Dénes
511c0123a2 Merge 'Add compaction module to task manager' from Aleksandra Martyniuk
Introduces task manager's compaction module. That's an initial
part of integration of compaction with task manager.

When fully integrated, task manager will allow user to track compaction
operations, check status and progress of each individual one. It will help
with creating an asynchronous version of rest api that forces any compaction.

Currently, users can see with /task_manager/list_modules api call that
compaction is one of the modules accessible through task manager.
They won't get any additional information though, since compaction
tasks are not created yet.

A shared_ptr to compaction module is kept in compaction manager.

Closes #12635

* github.com:scylladb/scylladb:
  compaction: test: pass task_manager to compaction_manager in test environment
  compaction: create and register task manager's module for compaction
  tasks: add task_manager constructor without arguments
2023-02-06 09:25:05 +02:00
Botond Dénes
cdd8b0fa35 Merge 'SSTable set improvements' from Raphael "Raph" Carvalho
Makes sstable_set::all() interface robust, and introduces sstable_set::size() to avoid copies when retrieving set size.

Closes #12716

* github.com:scylladb/scylladb:
  treewide: Use new sstable_set::size() wherever possible
  sstables: Introduce sstable_set::size()
  sstables: Fix fragility of sstable_set::all() interface
2023-02-06 08:24:00 +02:00
Avi Kivity
f73e2c992f Merge 'Keep range tombstones with rows in memtables and cache' from Tomasz Grabiec
This series switches memtable and cache to use a new representation for mutation data,
called `mutation_partition_v2`. In this representation, range tombstone information is stored
in the same tree as rows, attached to row entries. Each entry has a new tombstone field,
which represents range tombstone part which applies to the interval between this entry and
the previous one. See docs/dev/mvcc.md for more details about the format.

The transient mutation object still uses the old model in order to avoid work needed to adapt
old code to the new model. It may also be a good idea to live with two models, since the
transient mutation has different requirements and thus different trade-offs can be made.
Transient mutation doesn't need to support eviction and strong exception guarantees,
so its algorithms and in-memory representation can be simpler.

This allows us to incrementally evict range tombstone information. Before this series,
range tombstones were accumulated and evicted only when the whole partition entry was evicted. This
could lead to inefficient use of cache memory.

Another advantage of the new representation is that reads don't have to lookup
range tombstone information in a different tree while reading. This leads to simpler
and more efficient readers.

There are several disadvantages too. Firstly, rows_entry is now larger by 16 bytes.
Secondly, update algorithms are more complex because they need to deoverlap range tombstone
information. Also, to handle preemption and provide strong exception guarantees, update
algorithms may need to allocate sentinel entries, which adds complexity and reduces performance.

The memtable reader was changed to use the same cursor implementation
which cache uses, for improved code reuse and reducing risk of bugs
due to discrepancy of algorithms which deal with MVCC.

Remaining work:
  - performance optimizations to apply_monotonically() to avoid regressions
  - performance testing
  - preemption support in apply_to_incomplete (cache update from memtable)

Fixes #2578
Fixes #3288
Fixes #10587

Closes #12048

* github.com:scylladb/scylladb:
  test: mvcc: Extend some scenarios with exhaustive consistency checks on eviction
  test: mvcc: Extract mvcc_container::allocate_in_region()
  row_cache, lru: Introduce evict_shallow()
  test: mvcc: Avoid copies of mutation under failure injection
  test: mvcc: Add missing logalloc::reclaim_lock to test_apply_is_atomic
  mutation_partition_v2: Avoid full scan when applying mutation to non-evictable
  Pass is_evictable to apply()
  tests: mutation_partition_v2: Introduce test_external_memory_usage_v2 mirroring the test for v1
  tests: mutation: Fix test_external_memory_usage() to not measure mutation object footprint
  tests: mutation_partition_v2: Add test for exception safety of mutation merging
  tests: Add tests for the mutation_partition_v2 model
  mutation_partition_v2: Implement compact()
  cache_tracker: Extract insert(mutation_partition_v2&)
  mvcc, mutation_partition: Document guarantees in case merging succeeds
  mutation_partition_v2: Accept arbitrary preemption source in apply_monotonically()
  mutation_partition_v2: Simplify get_continuity()
  row_cache: Distinguish dummy insertion site in trace log
  db: Use mutation_partition_v2 in mvcc
  range_tombstone_change_merger: Introduce peek()
  readers: Extract range_tombstone_change_merger
  mvcc: partition_snapshot_row_cursor: Handle non-evictable snapshots
  mvcc: partition_snapshot_row_cursor: Support digest calculation
  mutation_partition_v2: Store range tombstones together with rows
  db: Introduce mutation_partition_v2
  doc: Introduce docs/dev/mvcc.md
  db: cache_tracker: Introduce insert() variant which positions before existing entry in the LRU
  db: Print range_tombstone bounds as position_in_partition
  test: memtable_test: Relax test_segment_migration_during_flush
  test: cache_flat_mutation_reader: Avoid timestamp clash
  test: cache_flat_mutation_reader_test: Use monotonic timestamps when inserting rows
  test: mvcc: Fix sporadic failures due to compact_for_compaction()
  test: lib: random_mutation_generator: Produce partition tombstone less often
  test: lib: random_utils: Introduce with_probability()
  test: lib: Improve error message in has_same_continuity()
  test: mvcc: mvcc_container: Avoid UB in tracker() getter when there is no tracker
  test: mvcc: Insert entries in the tracker
  test: mvcc_test: Do not set dummy::no on non-clustering rows
  mutation_partition: Print full position in error report in append_clustered_row()
  db: mutation_cleaner: Extract make_region_space_guard()
  position_in_partition: Optimize equality check
  mvcc: Fix version merging state resetting
  mutation_partition: apply_resume: Mark operator bool() as explicit
2023-02-05 22:33:10 +02:00
Pavel Emelyanov
d021aaf34d system_keysace: De-static calls that update view-building tables
There's a bunch of them used by mainly view_builder and also by the API
and storage_service. All use global qctx to make its job, now when the
callers have main-local sharded<system_keysace> references they can be
made non-static.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-02-03 21:56:54 +03:00