When we added zstd (f14e6e73bb), we used the static library
as we used some experimental APIs. However, now the dynamic
library works, so apparently the experimenal API is now standard.
Switch to the dynamic library. It doesn't improve anything, but it
aligns with how we do things.
Closes#12902
`int_range::make_singular()` accepts a single `int` as its parameter,
so there is no need to brace the paramter with `{}`. this helps to silence
the warning from Clang, like:
```
/home/kefu/dev/scylladb/test/perf/perf_fast_forward.cc:1396:63: error: braces around scalar initializer [-Werror,-Wbraced-scalar-init]
check_no_disk_reads(test(int_range::make_singular({100}))),
^~~~~
```
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#12903
Fixes https://github.com/scylladb/scylla-docs/issues/4140
This PR adds a new Knowledge Base article about improved garbage collection in ICS. It's based on the document created by @raphaelsc https://docs.google.com/document/d/1fA7uBcN9tgxeHwCbWftPJz071dlhucoOYO1-KJeOG8I/edit?usp=sharing.
@raphaelsc Could you review it? I've made some improvements to the language and text organization, but I didn't add or remove any content, so it should be a quick review.
@tzach requested a diagram, but we can add it later. It would be great to have this content published asap.
Closes#12792
* github.com:scylladb/scylladb:
doc: add the new KB to the list of topics
doc: add a new KB article about timbstone garbage collection in ICS
Task ttl can be set with task manager test api, which is disabled
in release mode.
Move get_and_update_ttl from task manager test api to task manager
api, so that it can be used in release mode.
Closes#12894
The developer documentation from `building.md` suggested to run unit tests with `./tools/toolchain/dbuild test` command, however this command only invokes `test` bash tool, which immediately returns with status `1`:
```
[piotrs@new-host scylladb]$ ./tools/toolchain/dbuild test
[piotrs@new-host scylladb]$ echo $?
1
```
This was probably unintended mistake and what author really meant was invoking `dbuild ninja test`.
Closes#12890
as `my_result_collector` has virtual function, and its dtor is not
marked virtual, Clang complains. let's mark its base class virtual
to be on the safe side.
```
/home/kefu/.local/bin/../lib/gcc/x86_64-pc-linux-gnu/13.0.1/../../../../include/c++/13.0.1/bits/unique_ptr.h:100:2: error: delete called on non-final 'my_result_collector' that has virtual functions but non-virtual destructor [-Werror,-Wdelete-non-abstract-non-virtual-dtor]
delete __ptr;
^
/home/kefu/.local/bin/../lib/gcc/x86_64-pc-linux-gnu/13.0.1/../../../../include/c++/13.0.1/bits/unique_ptr.h:405:4: note: in instantiation of member function 'std::default_delete<my_result_collector>::operator()' requested here
get_deleter()(std::move(__ptr));
^
/home/kefu/dev/scylladb/db/virtual_table.cc:134:25: note: in instantiation of member function 'std::unique_ptr<my_result_collector>::~unique_ptr' requested here
auto consumer = std::make_unique<my_result_collector>(s, permit, &pr, std::move(reader_and_handle.second));
^
```
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#12879
The former is a convenience wrapper over the latter. There's no real benefit in using it, but having two test_env-s is worse than just one.
Closes#12794
* github.com:scylladb/scylladb:
sstable_utils: Move the test_setup to perf/
sstable_utils: Remove unused wrappers over test_env
sstable_test: Open-code do_with_cloned_tmp_directory
sstable_test: Asynchronize statistics_rewrite case
tests: Replace test_setup::do_with_tmp_directory with test_env::do_with(_async)?
This patch adds a reproducer for the bug described in issue #7964 -
The restriction `where k=1 and c=2` (when k,c are the key columns)
returns (at most) a single row so doesn't need ALLOW FILTERING,
but if we add a third restriction, say `v=2`, this still processes
at most a single row so doesn't need ALLOW FILTERING - and both
Scylla and Cassandra get it wrong - so it's marked with both xfail
and cassandra_bug.
The patch also adds another test that for longer partition slices,
e.g., `where k=1 and c>2`, although the slice itself doesn't need
filtering, if we add `v=2` here we suddenly do need ALLOW FILTERING,
because the slice itself may be a large number of rows, and adding
`v=2` may restrict it to just a few results. This test passes
on both Scylla and Cassandra.
Issue #7964 mentioned these scenarios and even had some example code,
but we never added it to the test suite, so we finally do it now.
Refs #7964
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#12850
There are two methods to mess with compaction history -- update and get. The former had been patched to use local system-keyspace instance by 907fd2d3 (system_keyspace: De-static compaction history update) now it's time for the latter (spoiler: it's only used by the API handler)
Closes#12889
* github.com:scylladb/scylladb:
system_keyspace; Make get_compaction_history non static and drop qctx
api, compaction_manager: Get compaction history via manager
system_keyspace: Move compaction_history_entry to namespace scope
Tests of each module that is integrated with task manager use
calls to task manager api. Boilerplate to call, check status, and
get result may be reduced using functions.
task_manager_utils.py contains wrappers for task manager api
calls and helpers that may be reused by different tests.
Closes#12844
* github.com:scylladb/scylladb:
test: use functions from task_manager_utils.py in test_task_manager.py
test: add task_manager_utils.py
as an abstract base class `output_writer` is inherited by both
`json_output_writer` and `text_output_writer`. and `output_manager`
manages the lifecycles of used writers using
`std::unique_ptr<output_writer>`.
before this change, the dtor of `output_writer` is not marked as
virtual, so when its dtor is invoked, what gets called is the base
class's dtor. but the dtor of `json_output_writer` is non-trivial
in the sense that this class is aggregated by a bunch of member
variables. if we don't invoke its dtor when destroying this object,
leakage is expected.
so, in this change, the dtor of `output_writer` is marked as virtual,
this makes all of its derived classes' dtor virtual. and the right
dtor is always called.
test/perf is only designed for testing, and not used in production,
also, this feature was recently integrated into scylla executable in
228ccdc1c7.
so there is no need to backport this change.
change should also silence the warning from Clang 17:
```
/home/kefu/.local/bin/../lib/gcc/x86_64-pc-linux-gnu/13.0.1/../../../../include/c++/13.0.1/bits/unique_ptr.h:100:2: error: delete called on 'output_writer' that is abstract but has non-virtual destructor [-Werror,-Wdelete-abstract-non-virtual-dtor]
delete __ptr;
^
/home/kefu/.local/bin/../lib/gcc/x86_64-pc-linux-gnu/13.0.1/../../../../include/c++/13.0.1/bits/unique_ptr.h:405:4: note: in instantiation of member function 'std::default_delete<output_writer>::operator()' requested here
get_deleter()(std::move(__ptr));
^
/home/kefu/.local/bin/../lib/gcc/x86_64-pc-linux-gnu/13.0.1/../../../../include/c++/13.0.1/bits/stl_construct.h:88:15: note: in instantiation of member function 'std::unique_ptr<output_writer>::~unique_ptr' requested here
__location->~_Tp();
^
```
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#12888
This patch adds a cql-pytest test for an old secondary-index bug
that was described three years ago in issue #5823. cql-pytest makes
it easy to run the same test against different versions of Scylla,
and it was used to check that the bug existed in Scylla 2.3.0 but
was gone by 2.3.5, and also not present in master or in 2021.1.
A bit about the bug itself:
A secondary index is useful for equality restrictions (a=2) but can't be
used for inequality restrictions (a>=2). In Scylla 3.2.0 we used to have a
bug that because the restriction a>=2 couldn't be used through the index,
it was ignored completely. This is of course a mistake.
Refs #5823
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#12856
before this change, `seastar_memory_segment_store_backend`
is class with virtual method, but it does not have a virtual
dtor. but we do use a unique_ptr<segment_store_backend> to
manage the lifecycle of an intance of its derived class.
to enable the compiler to call the right dtor, we should
mark the base class's dtor as virtual. this should address
following warings from Clang-17:
```
/home/kefu/.local/bin/../lib/gcc/x86_64-pc-linux-gnu/13.0.1/../../../../include/c++/13.0.1/bits/unique_ptr.h:100:2: error: delete called on non-final 'logalloc::seastar_memory_segment_store_backend' that has virtual functions but non-virtual destructor [-Werror,-Wdelete-non-abstract-non-virtual-dtor]
delete __ptr;
^
/home/kefu/.local/bin/../lib/gcc/x86_64-pc-linux-gnu/13.0.1/../../../../include/c++/13.0.1/bits/unique_ptr.h:405:4: note: in instantiation of member function 'std::default_delete<logalloc::seastar_memory_segment_store_backend>::operator()' requested here
get_deleter()(std::move(__ptr));
^
/home/kefu/dev/scylladb/utils/logalloc.cc:812:20: note: in instantiation of member function 'std::unique_ptr<logalloc::seastar_memory_segment_store_backend>::~unique_ptr' requested here
: _backend(std::make_unique<seastar_memory_segment_store_backend>())
^
```
and
```
/home/kefu/.local/bin/../lib/gcc/x86_64-pc-linux-gnu/13.0.1/../../../../include/c++/13.0.1/bits/unique_ptr.h:100:2: error: delete called on 'logalloc::segment_store_backend' that is abstract but has non-virtual destructor [-Werror,-Wdelete-abstract-non-virtual-dtor]
delete __ptr;
^
/home/kefu/.local/bin/../lib/gcc/x86_64-pc-linux-gnu/13.0.1/../../../../include/c++/13.0.1/bits/unique_ptr.h:405:4: note: in instantiation of member function 'std::default_delete<logalloc::segment_store_backend>::operator()' requested here
get_deleter()(std::move(__ptr));
^
/home/kefu/dev/scylladb/utils/logalloc.cc:811:5: note: in instantiation of member function 'std::unique_ptr<logalloc::segment_store_backend>::~unique_ptr' requested here
contiguous_memory_segment_store()
^
```
Fixes#12872
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#12873
Since 5c0f9a8180 ("mutation_partition: Switch cache of
rows onto B-tree") it's no longer in use, except in some
performance test, so remove it.
Although scylla-gdb.py is sometimes used with older releases,
it's so outdated we can remove it from there too.
Closes#12868
we should never return a reference to local variable.
so in this change, a reference to a static variable is returned
instead. this should address following warning from Clang 17:
```
/home/kefu/dev/scylladb/tools/schema_loader.cc:146:16: error: returning reference to local temporary object [-Werror,-Wreturn-stack-address]
return {};
^~
```
Fixes#12875
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#12876
Now the call is done via the system_keyspace instance, so it can be
unmarked static and can use the local query processor instead of global
qctx.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Right now the API handler directly calls static method from system
keyspace. Patching it to call compaction manager instead will let the
latter use on-board plugged system keyspace for that. If the system
keyspace is not plugged, it means early boot or late shutdown, not a
good time to get compaction history anyway.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
We currently configure only TimeoutStartSec, but probably it's not
enough to prevent coredump timeout, since TimeoutStartSec is maximum
waiting time for service startup, and there is another directive to
specify maximum service running time (RuntimeMaxSec).
To fix the problem, we should specify RunTimeMaxSec and TimeoutSec (it
configures both TimeoutStartSec and TimeoutStopSec).
Fixes#5430Closes#12757
- date: drop implicitly generated ctor
- date: use std::in_range() to check for invalid year
Closes#12878
* github.com:scylladb/scylladb:
date: use std::in_range() to check for invalid year
date: drop implicitly generated ctor
Commit 1365e2f13e (gms: feature_service: re-enable features on node
startup) re-enabled features on feature service very early, so that on
boot a node sees its "correct" features state before it starts loading
system tables and replaying commitlog.
However, checking features happens on all shards independently, so
re-enabling should also happen on all shards.
One faced problem is in extract_scylla_specific_keyspace_info(). This
helper is used when loading non-system keyspace to read scylla-specific
keyspace options. The helper is called on all shards and on all-but-zero
it evaluates the checked SCYLLA_KEYSPACES feature to false leaving the
specific data empty. As the result, different shards have different view
of keyspaces' configuration.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closes#12881
because its base class of `writer_impl` has a member variable
`_validator`, which has its copy ctor deleted. let's just
drop the defaulted move ctor, as compiler is not able to
generate one for us.
```
/home/kefu/dev/scylladb/sstables/mx/writer.cc:805:5: error: explicitly defaulted move constructor is implicitly deleted [-Werror,-Wdefaulted-function-deleted]
writer(writer&& o) = default;
^
/home/kefu/dev/scylladb/sstables/mx/writer.cc:528:16: note: move constructor of 'writer' is implicitly deleted because base class 'sstable_writer::writer_impl' has a deleted move constructor
class writer : public sstable_writer::writer_impl {
^
/home/kefu/dev/scylladb/sstables/writer_impl.hh:29:48: note: copy constructor of 'writer_impl' is implicitly deleted because field '_validator' has a deleted copy constructor
mutation_fragment_stream_validating_filter _validator;
^
/home/kefu/dev/scylladb/mutation/mutation_fragment_stream_validator.hh:188:5: note: 'mutation_fragment_stream_validating_filter' has been explicitly marked deleted here
mutation_fragment_stream_validating_filter(const mutation_fragment_stream_validating_filter&) = delete;
^
```
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#12877
these warnings are found by Clang-17 after removing
`-Wno-unused-lambda-capture` and '-Wno-unused-variable' from
the list of disabled warnings in `configure.py`.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
There's the distribtued_loader::populate_column_family() helper that manages sstables on their way towards table on boot. The method naturally belongs the the table_population_metadata -- a helper class that in fact prepares the ground for the method in question.
This PR moves the method into metadata class and removes whole lot of extra alias-references and private-fields exporting methods from it. Also it keeps start_subdir and populate_c._f. logic close to each other and relaxes several excessive checks from them.
Closes#12847
* github.com:scylladb/scylladb:
distributed_loader: Rename table_population_metadata
distributed_loader: Dont check for directory presense twice
distributed_loader: Move populate calls into metadata.start()
distributed_loader: Remove local aliases and exporters
distributed_loader: Move populate_column_family() into population meta
It used to be just metadata by providing the meta for population, now it
does the population by itself, so rename it.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Both start_subdir() and populate_subdir() check for the directory to
exist with explicit file_exists() check. That's excessive, if the
directory wasn't there in the former call, the latter can get this by
checking the _sstable_directories map.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This makes the metadata class export even shorter API, keeps the three
sub-directories scanned in one place and allows removing the zero-shard
assertion.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
column_condition is an LWT-specific boolean expression construct, but
recent work allowed it to be re-expressed in terms of generic expressions.
This series completes the work and eliminates the column_condition classes
and source file. Furthermore, a statement's IF clause is represented as a
single expression, rather than a vector of per-column conditions.
Closes#12597
* github.com:scylladb/scylladb:
cql3: modification_statement: unwrap unnecessary boolean_factors() call
cql3: modification_statement: use single expression for conditions
cql3: modification_statment: fix lwt null equality rules mangling
cql3: broadcast tables: tighten checks on conditions
cql3: grammar: communicate LWT IF conditions to AST as a simple expression
cql3: column_condition: fold into modification_statement
cql3: column_condition: inline column_condition_applies_to into its only caller
cql3: column_condition: inline column_condition_collect_marker_specification into its only caller
cql3: column_condition: eliminate column_condition class
cql3: column_condition: move expression massaging to prepare()
cql3: grammar: make columnCondition production return an expression
cql3: grammar: eliminate duplication in LWT IF clause "IN (...)" vs "IN ?"
cql3: grammar: remove duplication around columnCondition scalar/collection variants
cql3: grammar: extract column references into a new production
cql3: column_condition: eliminate column_condition::raw
After previous patch all local alias references in
populate_column_family() are no longer requires. Neither are the
exporting calls from the table_population_metadata class.
Some non-obvious change is capturing 'this' instead of 'global_table' on
calls that are cross-shard. That's OK, table_population_metadata is not
sharded<> and is designed for cross-shard usage too.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This ownership change also requires the auto& = *this alias and extra
specification where to call reshard() and reshape() from.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
alternator headers are exposed to the target which links against it,
so let's expose them using the `target_include_directories()`.
also, `alternator` uses Seastar library and uses xxHash indirectly.
we should fix the latter by exposing the included header instead,
but for now, let's just link alternator directly to xxHash.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Scylla uses different build mode to customize the build for different
purposes. in this change, instead of having it in a python dictionary,
the customized settings are located in their own files, and loaded
on demand. we don't support multi-config generator yet.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
for better readability, and to silence following warning
from Clang 17:
```
/home/kefu/dev/scylladb/utils/date.h:5965:25: error: result of comparison of constant 9223372036854775807 with expression of type 'int' is always true [-Werror,-Wtautological-constant-out-of-range-compare]
Y <= static_cast<int64_t>(year::max())))
~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/kefu/dev/scylladb/utils/date.h:5964:57: error: result of comparison of constant -9223372036854775808 with expression of type 'int' is always true [-Werror,-Wtautological-constant-out-of-range-compare]
if (!(static_cast<int64_t>(year::min()) <= Y &&
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~
```
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
as one of its member variable does not have default constructor.
this silences following warning from Clang-17:
```
/home/kefu/dev/scylladb/utils/date.h:708:5: error: explicitly defaulted default constructor is implicitly deleted [-Werror,-Wdefaulted-function-deleted]
year_month_weekday() = default;
^
/home/kefu/dev/scylladb/utils/date.h:705:27: note: default constructor of 'year_month_weekday' is implicitly deleted because field 'wdi_' has no default constructor
date::weekday_indexed wdi_;
^
```
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Currently, we use two vectors for static and regular column conditions,
each element referring to a single column. There's a comment that keeping
them separate makes things simpler, but in fact we always treat both
equally (except in one case where we look at just the regular columns
and check that no static column conditions exist).
Simplify by storing just a single expression, which can be a conjunction
of mulitple column conditions.
add_condition() is renamed to analyze_condition(), since it now longers
adds to the vectors.
