Commit Graph

37583 Commits

Author SHA1 Message Date
Botond Dénes
586102b42e Merge 'readers: evictable_reader: don't accidentally consume the entire partition' from Kamil Braun
The evictable reader must ensure that each buffer fill makes forward progress, i.e. the last fragment in the buffer has a position larger than the last fragment from the previous buffer-fill. Otherwise, the reader could get stuck in an infinite loop between buffer fills, if the reader is evicted in-between.

The code guranteeing this forward progress had a bug: the comparison between the position after the last buffer-fill and the current last fragment position was done in the wrong direction.

So if the condition that we wanted to achieve was already true, we would continue filling the buffer until partition end which may lead to OOMs such as in #13491.

There was already a fix in this area to handle `partition_start` fragments correctly - #13563 - but it missed that the position comparison was done in the wrong order.

Fix the comparison and adjust one of the tests (added in #13563) to detect this case.

After the fix, the evictable reader starts generating some redundant (but expected) range tombstone change fragments since it's now being paused and resumed. For this we need to adjust mutation source tests which were a bit too specific. We modify `flat_mutation_reader_assertions` to squash the redundant `r_t_c`s.

Fixes #13491

Closes #14375

* github.com:scylladb/scylladb:
  readers: evictable_reader: don't accidentally consume the entire partition
  test: flat_mutation_reader_assertions: squash `r_t_c`s with the same position
2023-06-28 07:58:45 +03:00
Kefu Chai
c3d2f0cd81 script: add base36-uuid.py
this script provides a tool to decode a base36 encoded timeuuid
to the underlying msb and lsb bits, and to encode msb and lsb
to a string with base36.

Both scylla and Cassandra 4.x support this new SSTable identifier used
in SSTable names. like "nb-3fw2_0tj4_46w3k2cpidnirvjy7k-big-Data.db".
Since this is a new way to print timeuuid, and unlike the representation
defined by RFC4122, it is not straightforward to connect the the
in-memory representation (0x6636ac00da8411ec9abaf56e1443def0) to its
string representation of SSTable identifiers, like
"3fw2_0tj4_46w3k2cpidnirvjy7k". It would be handy to have this
tool to encode/decode the number/string for debugging purpose.

For more context on the new SSTable identifier, please
see
https://cassandra.apache.org/_/blog/Apache-Cassandra-4.1-New-SSTable-Identifiers.html
and https://issues.apache.org/jira/browse/CASSANDRA-17048

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes #14374
2023-06-27 16:56:31 +03:00
Kamil Braun
96bc78905d readers: evictable_reader: don't accidentally consume the entire partition
The evictable reader must ensure that each buffer fill makes forward
progress, i.e. the last fragment in the buffer has a position larger
than the last fragment from the previous buffer-fill. Otherwise, the
reader could get stuck in an infinite loop between buffer fills, if the
reader is evicted in-between.

The code guranteeing this forward progress had a bug: the comparison
between the position after the last buffer-fill and the current
last fragment position was done in the wrong direction.

So if the condition that we wanted to achieve was already true, we would
continue filling the buffer until partition end which may lead to OOMs
such as in #13491.

There was already a fix in this area to handle `partition_start`
fragments correctly - #13563 - but it missed that the position
comparison was done in the wrong order.

Fix the comparison and adjust one of the tests (added in #13563) to
detect this case.

Fixes #13491
2023-06-27 14:37:29 +02:00
Kamil Braun
5800ce8ddd test: flat_mutation_reader_assertions: squash r_t_cs with the same position
test_range_tombstones_v2 is too strict for this reader -- it expects a
particular sequence of `range_tombstone_change`s, but
multishard_combining_reader, when tested with a small buffer, may
generate -- as expected -- additional (redundant) range tombstone change
pairs (end+start).

Currently we don't observe these redundant fragments due to a bug in
`evictable_reader_v2` but they start appearing once we fix the bug and
the test must be prepared first.

To prepare the test, modify `flat_reader_assertions_v2` so it squashes
redundant range tombstone change pairs. This happens only in non-exact
mode.

Enable exact mode in `test_sstable_reversing_reader_random_schema` for
comparing two readers -- the squashing of `r_t_c`s may introduce an
artificial difference.
2023-06-27 14:37:25 +02:00
Botond Dénes
8d1dfbf0d9 Merge 'Fixing broken links to ScyllaDB University lessons, Scylla University…' from Guy Shtub
… -> ScyllaDB University

Closes #14385

* github.com:scylladb/scylladb:
  Update docs/operating-scylla/procedures/backup-restore/index.rst
  Fixing broken links to ScyllaDB University lessons, Scylla University -> ScyllaDB University
2023-06-27 13:05:46 +03:00
Avi Kivity
f86dd857ca Merge 'Certificate based authorization' from Calle Wilund
Fixes #10099

Adds the com.scylladb.auth.CertificateAuthenticator type. If set as authenticator, will extract roles from TLS authentication certificate (not wire cert - those are server side) subject, based on configurable regex.

Example:

scylla.yaml:

```
    authenticator: com.scylladb.auth.CertificateAuthenticator
    auth_superuser_name: <name>
    auth_certificate_role_query: CN=([^,\s]+)

    client_encryption_options:
      enabled: True
      certificate: <server cert>
      keyfile: <server key>
      truststore: <shared trust>
      require_client_auth: True
```
In a client, then use a certificate signed with the <shared trust> store as auth cert, with the common name <name>. I.e. for  qlsh set "usercert" and "userkey" to these certificate files.

No user/password needs to be sent, but role will be picked up from auth certificate. If none is present, the transport will reject the connection. If the certificate subject does not contain a recongnized role name (from config or set in tables) the authenticator mechanism will reject it.

Otherwise, connection becomes the role described.

To facilitate this, this also contains the addition of allowing setting super user name + salted passwd via command line/conf + some tweaks to SASL part of connection setup.

Closes #12214

* github.com:scylladb/scylladb:
  docs: Add documentation of certificate auth + auth_superuser_name
  auth: Add TLS certificate authenticator
  transport: Try to do early, transport based auth if possible
  auth: Allow for early (certificate/transport) authentication
  auth: Allow specifying initial superuser name + passwd (salted) in config
  roles-metadata: Coroutinuze some helpers
2023-06-27 12:52:14 +03:00
Guy Shtub
89bb690098 Update docs/operating-scylla/procedures/backup-restore/index.rst
Co-authored-by: Anna Stuchlik <37244380+annastuchlik@users.noreply.github.com>
2023-06-27 12:22:59 +03:00
Calle Wilund
00e5aec7ec docs: Add documentation of certificate auth + auth_superuser_name
Not great docs. But a start.
2023-06-27 07:38:50 +00:00
Benny Halevy
3ca0c6c0a5 compaction_manager: try_perform_cleanup: set owned_ranges_ptr with compaction disabled
Otherwise regular compaction can sneak in and
see !cs.sstables_requiring_cleanup.empty() with
cs.owned_ranges_ptr == nullptr and trigger
the internal error in `compaction_task_executor::compact_sstables`.

Fixes scylladb/scylladb#14296

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes #14297
2023-06-27 08:47:13 +03:00
Botond Dénes
f5e3b8df6d Merge 'Optimize creation of reader excluding staging for view building' from Raphael "Raph" Carvalho
View building from staging creates a reader from scratch (memtable
\+ sstables - staging) for every partition, in order to calculate
the diff between new staging data and data in base sstable set,
and then pushes the result into the view replicas.

perf shows that the reader creation is very expensive:
```
+   12.15%    10.75%  reactor-3        scylla             [.] lexicographical_tri_compare<compound_type<(allow_prefixes)0>::iterator, compound_type<(allow_prefixes)0>::iterator, legacy_compound_view<compound_type<(allow_prefixes)0> >::tri_comparator::operator()(managed_bytes_basic_view<(mutable_view)0>, managed_bytes
+   10.01%     9.99%  reactor-3        scylla             [.] boost::icl::is_empty<boost::icl::continuous_interval<compatible_ring_position_or_view, std::less> >
+    8.95%     8.94%  reactor-3        scylla             [.] legacy_compound_view<compound_type<(allow_prefixes)0> >::tri_comparator::operator()
+    7.29%     7.28%  reactor-3        scylla             [.] dht::ring_position_tri_compare
+    6.28%     6.27%  reactor-3        scylla             [.] dht::tri_compare
+    4.11%     3.52%  reactor-3        scylla             [.] boost::icl::interval_base_map<boost::icl::interval_map<compatible_ring_position_or_view, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sst+    4.09%     4.07%  reactor-3        scylla             [.] sstables::index_consume_entry_context<sstables::index_consumer>::process_state
+    3.46%     0.93%  reactor-3        scylla             [.] sstables::sstable_run::will_introduce_overlapping
+    2.53%     2.53%  reactor-3        libstdc++.so.6     [.] std::_Rb_tree_increment
+    2.45%     2.45%  reactor-3        scylla             [.] boost::icl::non_empty::exclusive_less<boost::icl::continuous_interval<compatible_ring_position_or_view, std::less> >
+    2.14%     2.13%  reactor-3        scylla             [.] boost::icl::exclusive_less<boost::icl::continuous_interval<compatible_ring_position_or_view, std::less> >
+    2.07%     2.07%  reactor-3        scylla             [.] logalloc::region_impl::free
+    2.06%     1.91%  reactor-3        scylla             [.] sstables::index_consumer::consume_entry(sstables::parsed_partition_index_entry&&)::{lambda()https://github.com/scylladb/scylladb/issues/1}::operator()() const::{lambda()https://github.com/scylladb/scylladb/issues/1}::operator()
+    2.04%     2.04%  reactor-3        scylla             [.] boost::icl::interval_base_map<boost::icl::interval_map<compatible_ring_position_or_view, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sst+    1.87%     0.00%  reactor-3        [kernel.kallsyms]  [k] entry_SYSCALL_64_after_hwframe
+    1.86%     0.00%  reactor-3        [kernel.kallsyms]  [k] do_syscall_64
+    1.39%     1.38%  reactor-3        libc.so.6          [.] __memcmp_avx2_movbe
+    1.37%     0.92%  reactor-3        scylla             [.] boost::icl::segmental::join_left<boost::icl::interval_map<compatible_ring_position_or_view, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::
+    1.34%     1.33%  reactor-3        scylla             [.] logalloc::region_impl::alloc_small
+    1.33%     1.33%  reactor-3        scylla             [.] seastar::memory::small_pool::add_more_objects
+    1.30%     0.35%  reactor-3        scylla             [.] seastar::reactor::do_run
+    1.29%     1.29%  reactor-3        scylla             [.] seastar::memory::allocate
+    1.19%     0.05%  reactor-3        libc.so.6          [.] syscall
+    1.16%     1.04%  reactor-3        scylla             [.] boost::icl::interval_base_map<boost::icl::interval_map<compatible_ring_position_or_view, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sst
+    1.07%     0.79%  reactor-3        scylla             [.] sstables::partitioned_sstable_set::insert

```
That shows some significant amount of work for inserting sstables
into the interval map and maintaining the sstable run (which sorts
fragments by first key and checks for overlapping).

The interval map is known for having issues with L0 sstables, as
it will have to be replicated almost to every single interval
stored by the map, causing terrible space and time complexity.
With enough L0 sstables, it can fall into quadratic behavior.

This overhead is fixed by not building a new fresh sstable set
when recreating the reader, but rather supplying a predicate
to sstable set that will filter out staging sstables when
creating either a single-key or range scan reader.

This could have another benefit over today's approach which
may incorrectly consider a staging sstable as non-staging, if
the staging sst wasn't included in the current batch for view
building.

With this improvement, view building was measured to be 3x faster.

from
`INFO  2023-06-16 12:36:40,014 [shard 0] view_update_generator - Processed keyspace1.standard1: 5 sstables in 963957ms = 50kB/s`

to
`INFO  2023-06-16 14:47:12,129 [shard 0] view_update_generator - Processed keyspace1.standard1: 5 sstables in 319899ms = 150kB/s`

Refs https://github.com/scylladb/scylladb/issues/14089.
Fixes scylladb/scylladb#14244.

Closes #14364

* github.com:scylladb/scylladb:
  table: Optimize creation of reader excluding staging for view building
  view_update_generator: Dump throughput and duration for view update from staging
  utils: Extract pretty printers into a header
2023-06-27 07:25:30 +03:00
Raphael S. Carvalho
1d8cb32a5d table: Optimize creation of reader excluding staging for view building
View building from staging creates a reader from scratch (memtable
+ sstables - staging) for every partition, in order to calculate
the diff between new staging data and data in base sstable set,
and then pushes the result into the view replicas.

perf shows that the reader creation is very expensive:
+   12.15%    10.75%  reactor-3        scylla             [.] lexicographical_tri_compare<compound_type<(allow_prefixes)0>::iterator, compound_type<(allow_prefixes)0>::iterator, legacy_compound_view<compound_type<(allow_prefixes)0> >::tri_comparator::operator()(managed_bytes_basic_view<(mutable_view)0>, managed_bytes
+   10.01%     9.99%  reactor-3        scylla             [.] boost::icl::is_empty<boost::icl::continuous_interval<compatible_ring_position_or_view, std::less> >
+    8.95%     8.94%  reactor-3        scylla             [.] legacy_compound_view<compound_type<(allow_prefixes)0> >::tri_comparator::operator()
+    7.29%     7.28%  reactor-3        scylla             [.] dht::ring_position_tri_compare
+    6.28%     6.27%  reactor-3        scylla             [.] dht::tri_compare
+    4.11%     3.52%  reactor-3        scylla             [.] boost::icl::interval_base_map<boost::icl::interval_map<compatible_ring_position_or_view, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sst+    4.09%     4.07%  reactor-3        scylla             [.] sstables::index_consume_entry_context<sstables::index_consumer>::process_state
+    3.46%     0.93%  reactor-3        scylla             [.] sstables::sstable_run::will_introduce_overlapping
+    2.53%     2.53%  reactor-3        libstdc++.so.6     [.] std::_Rb_tree_increment
+    2.45%     2.45%  reactor-3        scylla             [.] boost::icl::non_empty::exclusive_less<boost::icl::continuous_interval<compatible_ring_position_or_view, std::less> >
+    2.14%     2.13%  reactor-3        scylla             [.] boost::icl::exclusive_less<boost::icl::continuous_interval<compatible_ring_position_or_view, std::less> >
+    2.07%     2.07%  reactor-3        scylla             [.] logalloc::region_impl::free
+    2.06%     1.91%  reactor-3        scylla             [.] sstables::index_consumer::consume_entry(sstables::parsed_partition_index_entry&&)::{lambda()#1}::operator()() const::{lambda()#1}::operator()
+    2.04%     2.04%  reactor-3        scylla             [.] boost::icl::interval_base_map<boost::icl::interval_map<compatible_ring_position_or_view, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sst+    1.87%     0.00%  reactor-3        [kernel.kallsyms]  [k] entry_SYSCALL_64_after_hwframe
+    1.86%     0.00%  reactor-3        [kernel.kallsyms]  [k] do_syscall_64
+    1.39%     1.38%  reactor-3        libc.so.6          [.] __memcmp_avx2_movbe
+    1.37%     0.92%  reactor-3        scylla             [.] boost::icl::segmental::join_left<boost::icl::interval_map<compatible_ring_position_or_view, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::
+    1.34%     1.33%  reactor-3        scylla             [.] logalloc::region_impl::alloc_small
+    1.33%     1.33%  reactor-3        scylla             [.] seastar::memory::small_pool::add_more_objects
+    1.30%     0.35%  reactor-3        scylla             [.] seastar::reactor::do_run
+    1.29%     1.29%  reactor-3        scylla             [.] seastar::memory::allocate
+    1.19%     0.05%  reactor-3        libc.so.6          [.] syscall
+    1.16%     1.04%  reactor-3        scylla             [.] boost::icl::interval_base_map<boost::icl::interval_map<compatible_ring_position_or_view, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sst
+    1.07%     0.79%  reactor-3        scylla             [.] sstables::partitioned_sstable_set::insert

That shows some significant amount of work for inserting sstables
into the interval map and maintaining the sstable run (which sorts
fragments by first key and checks for overlapping).

The interval map is known for having issues with L0 sstables, as
it will have to be replicated almost to every single interval
stored by the map, causing terrible space and time complexity.
With enough L0 sstables, it can fall into quadratic behavior.

This overhead is fixed by not building a new fresh sstable set
when recreating the reader, but rather supplying a predicate
to sstable set that will filter out staging sstables when
creating either a single-key or range scan reader.

This could have another benefit over today's approach which
may incorrectly consider a staging sstable as non-staging, if
the staging sst wasn't included in the current batch for view
building.

With this improvement, view building was measured to be 3x faster.

from
INFO  2023-06-16 12:36:40,014 [shard 0] view_update_generator - Processed keyspace1.standard1: 5 sstables in 963957ms = 50kB/s

to
INFO  2023-06-16 14:47:12,129 [shard 0] view_update_generator - Processed keyspace1.standard1: 5 sstables in 319899ms = 150kB/s

Refs #14089.
Fixes #14244.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2023-06-26 22:30:39 -03:00
Raphael S. Carvalho
1ff8645eaa view_update_generator: Dump throughput and duration for view update from staging
Very helpful for user to understand how fast view update generation
is processing the staging sstables. Today, logs are completely
silent on that. It's not uncommon for operators to peek into
staging dir and deduce the throughput based on removal of files,
which is terrible.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2023-06-26 21:58:23 -03:00
Raphael S. Carvalho
83c70ac04f utils: Extract pretty printers into a header
Can be easily reused elsewhere.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2023-06-26 21:58:20 -03:00
Benny Halevy
9231a6c480 cql-pytest: test_using_timestamp: increase ttl
It seems like the current 1-second TTL is too
small for debug build on aarch64 as seen in
https://jenkins.scylladb.com/job/scylla-master/job/build/1513/artifact/testlog/aarch64/debug/cql-pytest.test_using_timestamp.1.log
```
            k = unique_key_int()
            cql.execute(f"INSERT INTO {table} (k, v) VALUES ({k}, {v1}) USING TIMESTAMP {ts} and TTL 1")
            cql.execute(f"INSERT INTO {table} (k, v) VALUES ({k}, {v2}) USING TIMESTAMP {ts}")
>           assert_value(k, v1)

test/cql-pytest/test_using_timestamp.py:140:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

k = 10, expected = 2

    def assert_value(k, expected):
        select = f"SELECT k, v FROM {table} WHERE k = {k}"
        res = list(cql.execute(select))
>       assert len(res) == 1
E       assert 0 == 1
E        +  where 0 = len([])
```

Increase the TTL used to write data to de-flake the test
on slow machines running debug build.

Ref #14182

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes #14396
2023-06-26 21:35:31 +03:00
Kefu Chai
fb05fddd7d build: build with -O0 if Clang >= 16 is used
to workaround https://github.com/llvm/llvm-project/issues/62842,
per the test this issue only surfaces when compiling the tree
with
ae7bf2b80b
which is included in Clang version 16, and the issue disappears
when the tree is compiled with -O0.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes #14391
2023-06-26 18:55:10 +03:00
Calle Wilund
a3db540142 auth: Add TLS certificate authenticator
Fixes #10099

Adds the com.scylladb.auth.CertificateAuthenticator type. If set as authenticator,
will extract roles from TLS authentication certificate (not wire cert - those are
server side) subject, based on configurable regex.

Example:

scylla.yaml:

authenticator: com.scylladb.auth.CertificateAuthenticator
auth_superuser_name: <name>
auth_certificate_role_queries:
	- source: SUBJECT
	  query: CN=([^,\s]+)

client_encryption_options:
  enabled: True
  certificate: <server cert>
  keyfile: <server key>
  truststore: <shared trust>
  require_client_auth: True

In a client, then use a certificate signed with the <shared trust>
store as auth cert, with the common name <name>. I.e. for cqlsh
set "usercert" and "userkey" to these certificate files.

No user/password needs to be sent, but role will be picked up
from auth certificate. If none is present, the transport will
reject the connection. If the certificate subject does not
contain a recongnized role name (from config or set in tables)
the authenticator mechanism will reject it.

Otherwise, connection becomes the role described.
2023-06-26 15:00:21 +00:00
Calle Wilund
20e9619bb1 transport: Try to do early, transport based auth if possible
Bypassing the need for an AUTH message+response. I.e. do auth _without_ client having
login specified.
2023-06-26 15:00:21 +00:00
Calle Wilund
a4b13febde auth: Allow for early (certificate/transport) authentication
Preparing for new authenticators. Hint hint.
2023-06-26 15:00:20 +00:00
Calle Wilund
69217662bd auth: Allow specifying initial superuser name + passwd (salted) in config
Instead of locking this to "cassandra:cassandra", allow setting in scylla.yaml
or commandline. Note that config values become redundant as soon as auth tables
are initialized.
2023-06-26 15:00:20 +00:00
Calle Wilund
3638849e63 roles-metadata: Coroutinuze some helpers
To make it easier to add options/features
2023-06-26 15:00:20 +00:00
Alexey Novikov
ca4e7f91c6 compact and remove expired rows from cache on read
when read from cache compact and expire row tombstones
remove expired empty rows from cache
do not expire range tombstones in this patch

Refs #2252, #6033

Closes #12917
2023-06-26 15:29:01 +02:00
Botond Dénes
b23361977b Merge 'Compaction reshape tasks' from Aleksandra Martyniuk
Task manager's tasks covering resharding compaction
on top and shard level.

Closes #14112

* github.com:scylladb/scylladb:
  test: extend test_compaction_task.py to test reshaping compaction
  compaction: move reshape function to shard_reshaping_table_compaction_task_impl::run()
  compaction: add shard_reshaping_compaction_task_impl
  replica: delete unused function
  compaction: add table_reshaping_compaction_task_impl
  compaction: copy reshape to task_manager_module.cc
  compaction: add reshaping_compaction_task_impl
2023-06-26 11:56:07 +03:00
Alejo Sanchez
4999cbc1cf test/boost/cql_functions_test: split long running tests
Split long running test_aggregate_functions to one case per type.

This allows test.py to run them in parallel.

Before this it would take 18 minutes to run in debug mode. Afterwards
each case takes 30-45 seconds.

Refs #13905

Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>

Closes #14368
2023-06-26 11:29:36 +03:00
Alejo Sanchez
8b1968cfbb test/boost/schema_changes_test: split long-running test
Split long running test test_schema_changes in 3 parts, one for each
writable_sstable_versions so it can be run in parallel by test.py.

Add static checks to alert if the array of types changed.

Original test takes around 24 minutes in debug mode, and each new split
test takes around 8 minutes.

Refs #13905

Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>

Closes #14367
2023-06-26 11:24:07 +03:00
Alejo Sanchez
633f026d63 test/boost/memtable_test: allow parallel run
Remove previous configuration blocking parallel run.

Test cases run fine in local debug.

Refs #13905

Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>

Closes #14369
2023-06-26 11:23:43 +03:00
Alejo Sanchez
3cbfd863eb test/boost/database_test: split long running tests
Split long running tests
test_database_with_data_in_sstables_is_a_mutation_source_plain and
test_database_with_data_in_sstables_is_a_mutation_source_reverse.

They run with x_log2_compaction_groups of 0 and 1, each one taking from
10 to 15 minutes each in debug mode, for a total of 28 and 22 minutes.

Split the test cases to run with 0 and 1, so test.py can run them in
parallel.

Refs #13905

Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>

Closes #14356
2023-06-26 11:20:27 +03:00
Takuya ASADA
c70a9cbffe scylla_fstrim_setup: start scylla-fstrim.timer on setup
Currently, scylla_fstrim_setup does not start scylla-fstrim.timer and
just enables it, so the timer starts only after rebooted.
This is incorrect behavior, we start start it during the setup.

Also, unmask is unnecessary for enabling the timer.

Fixes #14249

Closes #14252
2023-06-26 11:17:51 +03:00
Petr Gusev
1e851262f2 storage_proxy: handler responses, use pointers to default constructed values instead of nulls
The current Seastar RPC infrastructure lacks support
for null values in tuples in handler responses.
In this commit we add the make_default_rpc_tuple function,
which solves the problem by returning pointers to
default-constructed values for smart pointer types
rather than nulls.

The problem was introduced in this commit
 2d791a5ed4. The
 function `encode_replica_exception_for_rpc` used
 `default_tuple_maker` callback to create tuples
 containing exceptions. Callers returned pointers
 to default-constructed values in this callback,
 e.g. `foreign_ptr(make_lw_shared<reconcilable_result>())`.
 The commit changed this to just `SourceTuple{}`,
 which means nullptr for pointer types.

Fixes: #14282

Closes #14352
2023-06-26 11:10:38 +03:00
Anna Stuchlik
74fc69c825 doc: add Ubuntu 22 to 2021.1 OS support
Fixes https://github.com/scylladb/scylla-enterprise/issues/3036

This commit adds support for Ubuntu 22.04 to the list
of OSes supported by ScyllaDB Enterprise 2021.1.

This commit fixex a bug and must be backported to
branch-5.3 and branch-5.2.

Closes #14372
2023-06-26 10:41:43 +03:00
Aleksandra Martyniuk
197635b44b compaction: delete generation of new sequence number for table tasks
Compaction tasks covering table major, cleanup, offstrategy,
and upgrade sstables compaction inherit sequence number from their
parents. Thus they do not need to have a new sequence number
generated as it will be overwritten anyway.

Closes #14379
2023-06-26 10:36:10 +03:00
Guy Shtub
fa9df1b216 Fixing broken links to ScyllaDB University lessons, Scylla University -> ScyllaDB University 2023-06-25 09:09:49 +03:00
Aleksandra Martyniuk
b02a5fd184 test: extend test_compaction_task.py to test reshaping compaction 2023-06-23 16:22:53 +02:00
Aleksandra Martyniuk
f9a527b06d compaction: move reshape function to shard_reshaping_table_compaction_task_impl::run() 2023-06-23 16:22:53 +02:00
Aleksandra Martyniuk
1960904a72 compaction: add shard_reshaping_compaction_task_impl
shard_reshaping_compaction_task_impl covers reshaping compaction
on one shard.
2023-06-23 16:22:38 +02:00
Aleksandra Martyniuk
19ec5b4256 replica: delete unused function 2023-06-23 15:57:43 +02:00
Aleksandra Martyniuk
e3e2d6b886 compaction: add table_reshaping_compaction_task_impl 2023-06-23 15:57:37 +02:00
Aleksandra Martyniuk
dace5fb004 compaction: copy reshape to task_manager_module.cc
distributed_loader::reshape is copied to compaction/task_manager_module.cc
as it will be used in reshape compaction tasks.
2023-06-23 12:53:16 +02:00
Aleksandra Martyniuk
981a50e490 compaction: add reshaping_compaction_task_impl
reshaping_compaction_task_impl serves as a base class of all
concrete reshaping compaction task classes.
2023-06-23 12:53:15 +02:00
Kamil Braun
e6942d31d3 Merge 'query processor code cleanup' from Gleb
The series contains mostly cleanups for query processor and no functional
change. The last patch is a small cleanup for the storage_proxy.

* 'qp-cleanup' of https://github.com/gleb-cloudius/scylla:
  storage_proxy: remove unused variable
  client_state: co-routinise has_column_family_access function
  query_processor: get rid of internal_state and create individual query_satate for each request
  cql3: move validation::validate_column_family from client_state::has_column_family_access
  client_state: drop unneeded argument from has.*access functions
  cql3: move check for dropping cdc tables from auth to the drop statement code itself
  query_processor: co-routinise execute_prepared_without_checking_exception_message function
  query_processor: co-routinize execute_direct_without_checking_exception_message function
  cql3: remove empty statement::validate functions
  cql3: remove empty function validate_cluster_support
  cql3/statements: fix indentation and spurious white spaces
  query_processor: move statement::validate call into execute_with_params function
  query_processor: co-routinise execute_with_params function
  query_processor: execute statement::validate before each execution of internal query instead of only during prepare
  query_processor: get rid of shared internal_query_state
  query_processor: co-routinize execute_paged_internal function
  query_processor: co_routinize execute_batch_without_checking_exception_message function
  query_processor: co-routinize process_authorized_statement function
2023-06-23 10:32:57 +02:00
Kamil Braun
be5b61b870 Merge 'cql3: expr: break up expression.hh header' from Avi Kivity
It's very annoying to add a declaration to expression.hh and watch
the whole world get recompiled. Improve that by moving less-common
functions to a new header expr-utils.hh. Move the evaluation machinery
to a new header evaluate.hh. The remaining definitions in expression.hh
should not change as often, and thus cause less frequent recompiles.

Closes #14346

* github.com:scylladb/scylladb:
  cql3: expr: break up expression.hh header
  cql3: expr: restrictions.hh: protect against double inclusions
  cql3: constants: deinline
  cql3: statement_restrictions: deinline
  cql3: deinline operation::fill_prepare_context()
2023-06-23 10:19:28 +02:00
Nadav Har'El
0a1283c813 Merge 'cql3:statements:describe_statement: check pointer after casting to UDF/UDA' from Michał Jadwiszczak
There was a bug in describe_statement. If executing `DESC FUNCTION  <uda name>` or ` DESC AGGREGATE <udf name>`, Scylla was crashing because the function was found (`functions::find()` searches both UDFs and UDAs) but the function was bad and the pointer wasn't checked after cast.

Added a test for this.

Fixes: #14360

Closes #14332

* github.com:scylladb/scylladb:
  cql-pytest:test_describe: add test for filtering UDF and UDA
  cql3:statements:describe_statement: check pointer to UDF/UDA
2023-06-22 20:54:25 +03:00
Michał Jadwiszczak
d3d9a15505 cql-pytest:test_describe: add test for filtering UDF and UDA 2023-06-22 18:08:45 +02:00
Michał Jadwiszczak
d498451cdf cql3:statements:describe_statement: check pointer to UDF/UDA
While looking for specific UDF/UDA, result of
`functions::functions::find()` needs to be filtered out based on
function's type.

Fixes: #14360
2023-06-22 18:08:16 +02:00
Gleb Natapov
94fcba5662 storage_proxy: remove unused variable 2023-06-22 15:26:20 +03:00
Gleb Natapov
caee26ab4f client_state: co-routinise has_column_family_access function 2023-06-22 15:26:20 +03:00
Gleb Natapov
28f31bcbb1 query_processor: get rid of internal_state and create individual query_satate for each request
We want to put per-request field in query_satate, so make it unique for
each internal execution (it is unique for non internal once).
2023-06-22 15:26:20 +03:00
Avi Kivity
b858a4669d cql3: expr: break up expression.hh header
Adding a function declaration to expression.hh causes many
recompilations. Reduce that by:

 - moving some restrictions-related definitions to
   the existing expr/restrictions.hh
 - moving evaluation related names to a new header
   expr/evaluate.hh
 - move utilities to a new header
   expr/expr-utilities.hh

expression.hh contains only expression definitions and the most
basic and common helpers, like printing.
2023-06-22 14:21:03 +03:00
Avi Kivity
25c351a4f6 cql3: expr: restrictions.hh: protect against double inclusions
Add #pragma once. Right now it's safe as it only has declarations
(which can be repeated), but soon it will have a definition.
2023-06-22 14:19:43 +03:00
Avi Kivity
7302088274 cql3: constants: deinline
To reduce future header fan-in, deinline all non-trivial functions.
While these aer on the hot path, they can't be inlined anyway as they're
virtual, and they're quite heavy anyway.
2023-06-22 14:19:43 +03:00
Avi Kivity
6c0f8a73c5 cql3: statement_restrictions: deinline
Reduce future header fan-in by deinlining functions. These are
all on the prepare path.
2023-06-22 14:19:43 +03:00