This series adds two commands:
* scylla sstable-summary
* scylla sstable-index-cache
The former dumps the content of the sstable summary. This component is kept in memory in its entirety, so this can be easily done. The latter command dumps the content of the sstable index cache. This contains all the index-pages that are currently cached. The promoted index is not dumped yet and there is no indication of whether a given entry is in the LRU or not, but this already allows at seeing what pages are in the cache and what aren't.
Closes#10546
* github.com:scylladb/scylla:
scylla-gdb.py: add scylla sstable-index-cache command
scylla-gdb.py: add scylla sstable-summary command
test/scylla-gdb: add sstable fixture
scylla-gdb.py: make chunked_vector a proper container wrapper"
scylla-gdb.py: make small_vector a proper container wrapper"
scylla-gdb.py: add sstring container wrapper
scylla-gdb.py: add chunked_managed_vector container wrapper
scylla-gdb.py: add managed_vector container wrapper
scylla-gdb.py: std_variant: add workaround for clang template bug
scylla-gdb.py: add bplus_tree container wrapper
We have a check for whether we can use standard coroutines (in namespace
std) or the technical specification (in std::experimental), but it doesn't
work since Clang doesn't report the correct standard version. Use a
compiler versionspecific check, inspired by Seastar's check.
This allows building with clang 14.
Closes#10603
Let's say we have a query like:
```cql
INSERT INTO ks.t (list_column) VALUES (?);
```
And the driver sends a list with null inside as the bound value, something like `[1, 2, null, 4]`.
In such case we should throw `invalid_request_exception` because `nulls` are not allowed inside collections.
Currently when a query like this gets executed Scylla throws an ugly marshalling error.
This is because the validation code reads size of the next element, interprets it as an unsigned integer and tries to read this much.
In case of `null` element the size is `-1`, which when converted to unsigned `size_t` gives 18446744073709551615 and it fails to read this much.
This PR adds proper validation checks to make the error message better.
I also added some tests.
I originally tried to write them in python, but python driver really doesn't like sending invalid values.
Trying to send `[1, None, 2]` results in a list with empty value instead of null.
Trying to send `[1, UNSET_VALUE, 2]` Fails before query even leaves the driver.
Fixes#10580Closes#10599
* github.com:scylladb/scylla:
cql3: Add tests for null and unset inside collections
cql3: Add null and unset checks in collection validation
Add a bunch of tests that test what happens
when there is a null or unset value inside collections.
They are not allowed so every such attempt
should end with invalid_request_exception
with proper message.
I had to write a new function for collection serialization.
I tried to use data_value and its methods, but it's impossible
to create a data_value that represents an unset value.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
Reapply: "disable_auto_compaction: stop ongoing compactions"
This is a reapplication of a former commit
4affa801a5 which was reverted by
8e8dc2c930.
This commit is a fixed version of the original where a call to the
compaction_manager constructor accidentally issued (`compaction_manager()`)
instead a call to retrieve a compaction manager reference
(`get_compaction_manager()`), we don't use this function because it
doesn't exist anymore - it existed at the time the patch was written
bu was removed in 9066224cf4 later on,
instead, we just use the private table member _compaction_manager which refs
the compaction manager.
The explanation for the bad effect is probably that a `this` pointer
capture down the call chain, resulted in a use after free which had
an unknown effect on the system. (memory corruption at startup).
Test: unit (dev,debug)
write performance test as the one used to find the bug.
A screenshot of the performance test can be found at
https://github.com/scylladb/scylla/issues/10146/#issuecomment-1129578381
Fixes https://github.com/scylladb/scylla/issues/9313
Refs https://github.com/scylladb/scylla/issues/10146
For completeness, the original commit message was:
The api call disables new regular compaction jobs from starting
but it doesn't wait for ongoing compaction to stop and so it's
much less useful.
Returning after stopping regular compaction jobs and waiting
for them to stop guarantees that no regular compactions job are
running when nodetool disableautocompaction returns successfully.
Signed-off-by: Eliran Sinvani <eliransin@scylladb.com>
Closes#10597
* github.com:scylladb/scylla:
compaction_manager: Make invoking the empty constructor more explicit
Reapply: "disable_auto_compaction: stop ongoing compactions"
The compaction manager's empty constructor is supposed to be invoked
only in testing environment, however, it is easy to invoke it by mistake
from production code.
Here we add a more verbose constructor and making the default compaction
private, the verbose compiler need to be invoked with a tag
for_testing_tag, this will ensure that this constructor will be invoked
only when intended.
The unit tests were changed according to this new paradigm.
Tests: unit (dev)
Signed-off-by: Eliran Sinvani <eliransin@scylladb.com>
This is a reapplication of a former commit
4affa801a5 which was reverted by
8e8dc2c930.
This commit is a fixed version of the original where a call to the
compaction_manager constructor accidentally issued (`compaction_manager()`)
instead a call to retrieve a compaction manager reference
(`get_compaction_manager()`), we don't use this function because it
doesn't exist anymore - it existed at the time the patch was written
bu was removed in 9066224cf4 later on,
instead, we just use the private table member _compaction_manager which refs
the compaction manager.
The explanation for the bad effect is probably that a `this` pointer
capture down the call chain, resulted in a use after free which had
an unknown effect on the system. (memory corruption at startup).
Test: unit (dev,debug)
write performance test as the one used to find the bug.
A screenshot of the performance test can be found at
https://github.com/scylladb/scylla/issues/10146/#issuecomment-1129578381Fixes#9313
Refs #10146
For completeness, the original commit message was:
The api call disables new regular compaction jobs from starting
but it doesn't wait for ongoing compaction to stop and so it's
much less useful.
Returning after stopping regular compaction jobs and waiting
for them to stop guarantees that no regular compactions job are
running when nodetool disableautocompaction returns successfully.
Signed-off-by: Eliran Sinvani <eliransin@scylladb.com>
Validating a collection should ensure that there
are no null or unset values inside the collection.
The validation already fails in case of such values,
but it does so in an ugly way.
Length of null and unset value is negative but is
cast to unsigned size_t. Then it tries to read
a really large value and fails with marshalling error.
The new checks are a better way to handle this.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
This test started failing sporadically of late. This failure is seen
quite often in CI tests but is very hard to reproduce locally. The
problem seems to be timing related, as the same seeds that fail in CI
don't fail locally. This patch is a speculative fix. The test has a
single time-related components: `gc_clock::now()`. This is invoked in 4
different places during a single iteration, giving ample opportunity for
off-by-one errors to appear. Although there is no solid proof for this
being the problem, this is a good candidate. This patch replaces all
those different invocations, with a single one per test: this value is
then propagated to all places that need it.
Fixes: #10554
Marking the patch as a fix for the issue, if the problem re-surfaces
after this patch we'll re-poen it.
Closes#10589
Using traceback_with_variables module, generate more detail traceback
with variables into debug log.
This will help fixing bugs which is hard to reproduce.
Closes#10472
[avi: regenerate frozen toolchain]
Today, aborting a maintenance compaction like major, which is waiting for
its turn to run, can take lots of time because compaction manager will
only be able to bail out the task once it gets the "permit" from the
serialization mechanism, i.e. semaphore. Meaning that the command that
started the task will only complete after all this time waiting for
the "permit".
To allow a pending maintenance compaction to be quickly aborted, we
can use the abortable variant of get_units(). So when user submits an
abortion request, get_units() will be able to return earlier through
the abort exception.
Refs #10485.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Closes#10581
Before this change parser used to output instances of the `relation` class, which were later converted to `restrction`.
`relation` took care of initial processing such as preparing and some validation checks.
This PR aims to remove the `relation` class and perform it's functionality using only `expression`.
This is a step towards removing the legacy classes and converting all AST analysis to work on `expressions`.
Closes#10409
* github.com:scylladb/scylla:
cql3: Remove scalar from bind_variable_scalar_prepare_expression
cql3: expr: Remove shape_type from bind_variable
cql3: Remove prepare_expression_multi_column
cql3: Remove relation class
cql3: Add more tests for expr::printer
cql3: Make parser output expression for relations
cql3: expr: add printer for expression
cql3: expr: expr::to_restriction: Handle token relations
cql3: expr: expr::to_restriction: Handle multi column relations
cql3: expr: Add expr::to_restriction for single column relations
cql3: expr: Add prepare_binary_operator
cql3: expr: Change how prepare_expression handles bind_variable
clq3: expr: Add columns to expr::token struct
cql3: expr: Modify list_prepare_expression to handle lists of IN values
cql3: expr: Add expr::as_if for non-const expressions
Currently our error message on scylla_prepare says "Exception occurred
while creating perftune.yaml", even perftune.yaml is already generated,
and error occurred after that.
To describe error more correctly, add another error message after
perftune.yaml generated.
see scylladb/scylla-enterprise#2201
Closes#10575
shape_type was used in prepare_expression to differentiate
between a few cases and create the correct receivers.
This was used by the relation class.
Now creating the correct receiver has been delegated to the caller
of prepare_expression and all bind_variables can be handled
in the same simple way.
shape_type is not needed anymore.
Not having it is better because it simplifies things.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
This function was used by multi_column_relation.hh,
but now it isn't needed anymore.
The only way to prepare a bind_variable is now the standard prepare_expression.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
Functionality of the relation class has been replaced by
expr::to_restriction.
Relation and all classes deriving from it can now be removed.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
Now that parser outputs expressions
it's much easier to check whether
expression printer works correctly.
We can prepare a bunch of strings
which will be parsed and then printed
back to string.
Then we can compare those strings.
It's much easier than creating
expresions to print manually.
The only downside is that this tests
only unprepared version of expression,
so instead of column_value there will
be unresolved identifier, insted of constant
untyped_constant etc.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
Parser used to output the where clause as a vector of relations,
but now we can change it to a vector of expressions.
Cql.g needs to be modified to output expressions instead
of relations.
The WHERE clause is kept in a few places in the code that
need to be changed to vector<expression>.
Finally relation->to_restriction is replaced by expr::to_restriction
and the expressions are converted to restrictions where required.
The relation class isn't used anywhere now and can be removed.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
expression::printer is used to print CQL expressions
in a pretty way that allows them to be parsed back
to the same representation.
There is a bunch of things that need to be changed when
compared to the current implementation of opreatorr<<(expression)
to output something parsable.
column names should be printed without 'unresolved_identifier()'
and sometimes they need to be quoted to perserve case sensitivity.
I needed to write new code for printing constant values
because the current one did debug printing
(e.g. a set was printed as '1; 2; 3').
A list of IN values should be printed inside () intead of [],
but because it is internally represented as a list it is
by default printed with [].
To fix this a temporary tuple_constructor is created and printed.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
Implement converting token relations to expressions.
The code is mostly tekken from functions in token_relation.hh,
because we are replicating functionliaty of the functions called
token_relation::new_XX_restrictions.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
Implement converting multi column relations to expressions.
The code is mostly taken from functions in multi_column_relation.hh,
because we are replicating functionality of the functions called
multi_column_relation::new_XX_restriction.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
Add a function that will be used to convert expressions
received from the parser to restrictions.
Currently parser creates relations with expressions inside
and then those relations are converted to restrictions.
Once this function is implemented we will be able to skip
creating relations altogether and convert straight from
expression to restriction. This will allow us to remove
the relation class.
Further functionality will be implemented in the following commits.
This commit implements converting single column relations to expressions.
The code is mostly taken from functions in single_column_relation.hh,
because we are replicating functionality of the functions called
single_column_relation::new_XX_restriction.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
Add a function that allows to prepare
a binary_operator received from the parser.
It resolves columns on the LHS, calculates type of LHS,
and prepares RHS with the correct type.
It will be used by expr::to_restriction.
Some basic type checks are performed, but more throughout
checks will be required in expr::to_restriction to fully
validate a relation.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
The situation with preparing bind_variable is a bit strange,
there are four shapes of bind variables and receiver behaviour
is not in line with other types.
To prepare a bind_variable for a list of IN values for an int column
the current code requires us to pass a receiver of type int.
This is counterintuitive, to prepare a string we pass
a receiver with string type, so to prepare list<int> we should
pass a receiver of type list<int>, not just int.
This commit changes the behaviour in two ways:
- Shape of bind_variable doesn't matter anymore
- The bind_variable gets the receiver passed to prepare_expression,
no more list<receiver> magic.
Other variants of bind_variable_x_prepare_expression are not removed yet
because they are needed by prepare_expression_mutlti_column.
They will be removed later, along with bind_variable::shape_type.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
The expr::token struct is created when something
like token(p1, p2) occurs in the WHERE clause.
Currently expr::token doesn't keep columns passed
as arguemnts to the token function.
They weren't needed because token() validation
was done inside token_relation.
Now that we want to use only expressions
we need to have columns inside the token struct
and validate that those are the correct columns.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
The standard CQL list type doesn't allow for nulls inside the collection.
However lists of IN values are the exception where bind nullsare allowed,
for example in restrictions like: p IN (1, 2, null)
To be able to use list_prepare_expression with lists of IN values
a flag is added to specify whether nulls should be allowed.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
expr::as_if is our wrapper for std::get_if.
There was a version for const expression*,
but there weren't one for mutable expression*.
Add the mutable version,
it will be needed in the following commits.
Signed-off-by: cvybhu <jan.ciolek@scylladb.com>
Our documentation for SELECT,
https://docs.scylladb.com/getting-started/dml/#ordering-results
says that:
"The ORDER BY clause lets you select the order of the returned results.
It takes as argument a list of column names along with the order for
the column (ASC for ascendant and DESC for descendant,
**omitting the order being equivalent to ASC**)."
The test in this patch confirms that the last emphasized line is not
accurate - The default order for SELECT is the default order of the table
being read - NOT always ascending order. If the table was created with
descending WITH CLUSTERING ORDER BY, then a SELECT not specifying
an ORDER BY will get this descending order by default.
The test passes on both Scylla and Cassandra, demonstrating that this
behavior is expected and correct - regardless of what our docs say.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20220515115030.775813-1-nyh@scylladb.com>
When split from service/storage_service.hh (4ca2ae13) it accidentally
changed license. Change it back (since it does not contain
Apache derived code, constrain it to AGPL-3.0-or-later).
Closes#10572
The bytes_stream param is passed by value from `write_promoted_index`
(since 0d8463aba5)
causing an uneeded copy.
This can lead to OOM if the promoted index is extremely large.
Pass the bytes_ostream by reference instead to prevent this copy.
Fixes#10569
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Closes#10570
Current code already assumes (correctly), that shrink() does not
throw, otherwise we risk leaking memory allocated in get_ptr():
```
ts_value_lru_entry* new_lru_entry = Alloc().template allocate_object<ts_value_lru_entry>();
// Remove the least recently used items if map is too big.
shrink();
```
Let's be explicit and mark shrink() and a few helper methods
that it uses as noexcept. Ultimately they are all noexcept anyway,
because polymorphic allocator's deallocation routines don't throw,
and neither do boost intrusive list iterators.
Closes#10565
It turns out that there is a difference between how Scylla and Cassandra
handle multiple restrictions on the same column - for example "WHERE
c = 0 and c >0". Cassandra treats all such cases as invalid queries,
whereas Scylla *allows* them.
This test demonstrates this difference (it is marked "scylla_only"
because it's a Scylla-only feature), and also verifies that the results
of such queries on Scylla are correct - i.e., if the two restrictions
conflict the result is empty, and if the two restrictions overlap,
the result can be non-empty.
The test passes, verifying that although Scylla differs from Cassandra
on this, its behavior is correct.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20220512165107.644932-1-nyh@scylladb.com>
Recently Abseil started to ask to enable ABSL_PROPAGATE_CXX_STD,
warning that it will do so itself in the future. Do so, and
specify that we use C++20 to avoid inconsistencies.
Closes#10563
After fcb8d040 ("treewide: use Software Package Data Exchange
(SPDX) license identifiers"), many dual-licensed files were
left with empty comments on top. Remove them to avoid visual
noise.
Closes#10562
"Almost" because 2 uses of the v1 asserter remain (as they are deliberate).
Closes#10518
* github.com:scylladb/scylla:
tests: remove obsolete utility functions
tests: less trivial flat_reader_assertions{,_v2} conversions
tests: trivial flat_reader_assertions{,_v2} conversions
flat_mutation_reader_assertions_v2: improve range tombstone support
This reverts commit 4affa801a5.
In issue #10146 a write throughput drop of ~50% was reported, after
bisect it was found that the change that caused it was adding some
code to the table::disable_auto_compaction which stops ongoing
compactions and returning a future that resolves once all the compaction
tasks for a table, if any, were terminated. It turns out that this function
is used only at startup (and in REST api calls which are not used in the test)
in the distributed loader just before resharding and loading of
the sstable data. It is then reanabled after the resharding and loading
is done.
For still unknown reason, adding the extra logic of stopping ongoing
compactions made the write throughput drop to 50%.
Strangely enough this extra logic **should** (still unvalidated) not
have any side effects since no compactions for a table are supposed to
be running prior to loading it.
This regains the performance but also undo a change which eventually
should get in once we find the actual culprit.
Signed-off-by: Eliran Sinvani <eliransin@scylladb.com>
Closes#10559Reopens#9313.
We're hitting a unit test failure as in
https://jenkins.scylladb.com/view/master/job/scylla-master/job/build/1010/artifact/testlog/aarch64_dev/frozen_mutation_test.test_writing_and_reading_gently.918.log
```
unknown location(0): fatal error: in "test_writing_and_reading_gently": std::_Nested_exception<std::runtime_error>: frozen_mutation::unfreeze_gently(): failed unfreezing mutation pk{00801806000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000} of ks.cf
```
on aarch64 clang 12.0.1, in release and dev modes (but not debug).
This turned out to be a miscompilation
in `position_in_partition_view::for_key(cr.key())`
that returns a position_in_partition_view of the
clustering_key_prefix rvalue that cr.key() returns.
The latter is lost on aarch64 in release mode.
Keeping the key on the stack allows to safely pass
a view to it.
Fixes#10555
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
The main reason for adding rust dependency to scylla is the
wasmtime library, which is written in rust. Although there
exist c++ bindings, they don't expose all of its features,
so we want to do that ourselves using rust's cxx.
The patch also includes an example rust source to be used in
c++, and its example use in tests/boost/rust_test.
The usage of wasmtime has been slightly modified to avoid
duplicate symbol errors, but as a result of adding a Rust
dependency, it is going to be removed from `configure.py`
completely anyway
Signed-off-by: Wojciech Mitros <wojciech.mitros@scylladb.com>
Closes#10341
* github.com:scylladb/scylla:
docs: document rust
tests: add rust example
Only for reasons other than "no such KS", i.e. when the failure is
presumed transient and the batch in question is not deleted from
batchlog and will be retried in the future.
(Would info be more appropriate here than warning?)
Signed-off-by: Michael Livshin <michael.livshin@scylladb.com>
Closes#10556
Using Rust in Scylla is not intuitive, the doc explains the entire
process of adding new Rust source files to Scylla. What happens
during compilation is also explained.
Signed-off-by: Wojciech Mitros <wojciech.mitros@scylladb.com>
The patch includes an example rust source to be used in
c++, and its example use in tests/boost/rust_test.
Signed-off-by: Wojciech Mitros <wojciech.mitros@scylladb.com>
This series update debugging.md with:
- add an example .gdbinit file
- update recommendation for finding the relocatable packages using a build-id on http://backtrace.scylladb.com/Closes#10492
* github.com:scylladb/scylla:
docs: debugging.md: update instructions regarding backtrace.scylladb.com
docs: debugging.md: add a sample gdbinit file
Drop the section about non-relocatable packages. They are not a thing anymore.
Also tweaked the instructions for launching the toolchain container.
Closes#10539
* github.com:scylladb/scylla:
docs/debugging.md: adjust instructions for using the toolchain
docs/debugging.md: drop section about handling binaries from non-relocatable packages
