Commit Graph

44617 Commits

Author SHA1 Message Date
Marcin Maliszkiewicz
4630864b58 db: remove unused flush arg from do_merge_schema func 2024-09-23 12:01:36 +02:00
Marcin Maliszkiewicz
4cce9c8b5a db: remove unused read_arg_values function 2024-09-23 12:01:36 +02:00
Kefu Chai
2014d1c0cb cql3: drop workaround for castas_fctn_simple()
now that e13a584ab7
has been merged, and our toolchain is based on the fedora 40 on 20240710, which should include this
change.

so let's drop the workaround from 51d09e6a

Refs #18508
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20750
2024-09-22 19:59:10 +03:00
Kefu Chai
fdc8773278 test/scylla_gdb: get table::_schema raw pointer with lw_shared_ptr
This commit addresses an issue where accessing the raw pointer of the
schema instance within `table::_schema` using `table.schema._p` was
unreliable.

before this change, `_p` was of type `lw_shared_ptr_counter_base*`, a
type-erased smart pointer, preventing direct casting to the underlying
schema pointer. but we still cast it to `schema*` anyway. this led to
a gdb.MemoryError when dereferencing the deduced pointer:
but the type of `_p` is `lw_shared_ptr_counter_base*`, which is a
type erased smart pointer, and it cannot be casted directly to the
under pointer pointing to a `schema` instance. this results in:

```
Traceback (most recent call last):
  File "/home/avi/scylla/test/scylla_gdb/../../scylla-gdb.py", line 5554, in invoke
    self.print_key_type(seastar_lw_shared_ptr(schema['_clustering_key_type']).get().dereference(), 'clustering')
  File "/home/avi/scylla/test/scylla_gdb/../../scylla-gdb.py", line 5533, in print_key_type
    key_type = seastar_shared_ptr(key_type).get().dereference()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gdb.MemoryError: Cannot access memory at address 0x4000079656b0078
```

when we are dereferencing the raw pointer deduced this way.

in this change,

* we use the wrapper of `seastar_lw_shared_ptr` to safely obtain the
  raw pointer.
* reenable this test previously disabled by 3d781c4f

tested using
```console
$ SCYLLA=/home/kefu/dev/scylladb/master/build/release/scylla \
  test/scylla_gdb/run -o junit_suite_name=scylla_gdb test_misc.py::test_schema
```

on an up-to-date fedora 40 installation.

Refs 3d781c4f
Fixes scylladb/scylladb#20741
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20746
2024-09-22 18:30:16 +03:00
Avi Kivity
657848dcbb cql3: statement_restrictions, expr: move restrictions-related expression utilities out of expression.cc
Move all of the blatantly restriction-related expression utilities
to statement_restrictions.cc.

Some are so blatant as to include the word "restriction" in their name.
Others are just so specialized that they cannot be used for anything else.

The motivation is that further refactoring will be simplified if it can
happen within the same module, as there will not be a need to prove
it has no effect elsewhere.

Most of the declarations are made non-public (in .cc file) to limit
proliferation. A few are needed for tests or in select_statement.cc
and so are kept public.

Other than that, the only changes are namespace qualifications and
removal of a now-duplicate definition ("inclusive").

Closes scylladb/scylladb#20732
2024-09-22 11:00:51 +03:00
Avi Kivity
3d781c4fc8 Update frozen toolchain
* tools/java e505a6d3bb...5b0e274f12 (1):
  > Merge 'build.xml: install and use java-11 when building' from Kefu Chai

Updates to clang 18.1.8 + LLVM patch to match Fedora 40.

New optimized clang build generated and stored in

    https://devpkg.scylladb.com/clang/clang-18.1.8-x86_64.tar.gz
    https://devpkg.scylladb.com/clang/clang-18.1.8-aarch64.tar.gz

Due to the loss of the jmx submodule, we no longer install java-11-openjdk.
We add it in install-dependencies.sh here to compensate, pending a better
solution.

tools/java submodule updated to remove build failure where Java 8
was selected instead of Java 11.

The scylla_gdb test suite was disabled due to a regression in gdb 15,
which is brought in by the toolchain update [1].

[1] https://github.com/scylladb/scylladb/issues/20741.
2024-09-21 20:07:28 +03:00
Avi Kivity
cd861bc788 row_cache: coroutinize do_update()
do_with() makes the change a no-brainer, and besides, it's called once
per huge update.

Closes scylladb/scylladb#20735
2024-09-21 00:07:02 +02:00
Botond Dénes
488a372fdc tool/scylla-nodetool: status: reorder endpoint calls to match old nodetool
Old nodetool requested `/storage_service/tokens_endpoing` first, then
`/storage_service/host_id`, while the native nodetool did it in reverse
order. Most of the time this is inconsequential but there is an edge
case when a node's IP address is changed. This reversing of the order
results in unexpected behavior for tests, causing noise via flaky
tests.
Match the order of the old nodetool so that the native nodetool exhibits
the behavior expected by tests (and users too probably).

Fixes: scylladb/scylladb#18693

Closes scylladb/scylladb#20615
2024-09-20 15:07:16 +02:00
Avi Kivity
61d19e4464 Update tools/java submodule
* tools/java 0b4accdd5e...e505a6d3bb (1):
  > [C-S] Make it use DCAwareRoundRobinPolicy unless rack is provided
2024-09-20 14:49:21 +03:00
Pavel Emelyanov
b45891acd7 sstables: storage: Don't keep base directory in base class
This reverts commit 44bd183187 and
moves the base directory back on filesystem_storage. The mentioned
commit says

>     so we can use the base (table) directory for
>    e.g. pending_delete logs, in the next patch.

but "next patch" doesn't use it outside of the filesystem-storage
anyway.

This field doesn't make sense for S3 backend. Its "location" is not
location, but a key in the system.sstables, which should rather be
schema ID, not /var/lib/.../keyspace/table-uuid string.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>

Closes scylladb/scylladb#20642
2024-09-20 11:51:04 +03:00
Andrei Chekun
bd9a73c39b Add .idea folder to .gitignore
.idea directory used by JetBrains IDE's to store data about project
config

Closes scylladb/scylladb#20718
2024-09-20 11:49:41 +03:00
Tomasz Grabiec
8e047e8fff gdb: Add std::set wrapper
Allows accessing std::set fields from gdb, e.g.:

(gdb) python for e in std_set(_promoted_index._blocks): print(e)

Closes scylladb/scylladb#20650
2024-09-20 08:24:15 +03:00
Anna Stuchlik
5da7894f70 doc: move the install-jmx instructions to a common folder
This commit moves the install-jmx.rst file from the install-scylla folder
to the installation-common folder.

All the references to the moved document are updated.

This is a follow-up to https://github.com/scylladb/scylladb/pull/17969/

Closes scylladb/scylladb#20712
2024-09-20 00:36:32 +03:00
Nadav Har'El
3499c407f7 test: avoid silly "no_mode.1" labels when running tests outside test.py
For the benefit of running test.py inside CI, we recently added to
test/cql-pytest and test/alternator the knowledge of which "Scylla mode"
(--mode) and "run number" is running (--run_id), although these concepts
are alien to these two test frameworks (remember that those test frameworks
can also run tests against unknown versions of Scylla or even our competitors'
implementations).

One unfortunate result of this change is that now if you run a test by
using pytest directly (or test/*/run) instead of test.py, for example:

    $ cd test/alternator
    $ pytest --aws test_item.py::test_basic_string_put_and_get

The test's success or failure reports the ugly name

    test_item.py::test_basic_string_put_and_get.no_mode.1

This unnecessary "no_mode.1" come from the the default values for --mode
and --run_id, respectively. But there is no reason for these silly
defaults. In this patch we change these defaults to None, and when they
are None, they aren't tacked onto the test's name.

This patch shouldn't affect running tests through test.py, because
test.py always sets the --mode and --run_id options, and doesn't leave
them as the default.

Fixes #20512

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes scylladb/scylladb#20513
2024-09-20 00:36:32 +03:00
Avi Kivity
b015c85d31 Merge 'gms: inet_address: drop unused raw_addr method and modernize comperators' from Benny Halevy
Drop the unused `gms::inet_address::raw_addr` method
and modernize operator== and operator< as class methods

* Cleanup only, no backport needed

Closes scylladb/scylladb#20681

* github.com:scylladb/scylladb:
  gms: inet_address: modernize comparison operators
  gms: inet_address: drop unused raw_addr method
2024-09-20 00:36:32 +03:00
Piotr Dulikowski
7e7701d436 Merge 'cql3/statements/select_statement: SELECT ... USING SERVICE LEVEL' from Michał Jadwiszczak
Allow to specify service level used in select statement `SELECT ... USING SERVICE LEVEL sl_name`.
In OSS, this only affects statement's timeout.

In case both service level and timeout are specified `SELECT ... USING SERVICE LEVEL sl_name AND TIMEOUT 1h`, the timeout has higher priority as statement's timeout.

Fixes scylladb/scylladb#18471

Closes scylladb/scylladb#20523

* github.com:scylladb/scylladb:
  test/cql-pytest: add test for `SELECT ... USING SERVICE LEVEL`
  cql3/Cql.g: extend grammar to allow `SELECT ... USING SERVICE LEVEL`
  cql3/statements/select_statement: use service level timeout
  cql3/attributes: add service level name field
  qos/service_level_controller: add method to check if service level exists in cache
2024-09-19 18:19:23 +02:00
Pavel Emelyanov
bd720dd2da Merge 'cql3: statement_restrictions: adapt to functional style' from Avi Kivity
The statement_restrictions class started life in the object-oriented style - an
object that interacts with its environment via mutators and is observed via
observers.

This is however not suitable for its objective: to analyze the WHERE clause,
select a query plan, and partition the WHERE clause atoms to the various
parts demanded by the query plan (read_command and filters). Furthermore,
the object oriented style makes it hard to work with as you can only call some
observers after the related mutators were called.

Fix this by transforming the code info a more functional style: we call
a function that returns an immutable statement_restrictions object that
can only be observed. This makes it easier to further change in the future,
as changes will not have to consider interaction with the environment.

No backport as this is a refactoring

Closes scylladb/scylladb#20672

* github.com:scylladb/scylladb:
  cql3: statement_restrictions: use functional style
  cql3: statement_restrictions: calculate the index only once
  cql3: statement_restrictions: make it a const object
2024-09-19 18:18:28 +03:00
Kefu Chai
8cc9d783a0 sstables/sstable_directory: document components_lister::process()
for better maintainability.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20693
2024-09-19 18:11:31 +03:00
Kefu Chai
7985aa97b1 main, test: use seastar::handle_signal() instead
use `seastar::handle_signal()` instead of `reactor::handle_signal()`.

in a recent change in seastar (c3e826ad1197f2610138f3bcfaeb0b458f8fb799),
the later was marked as deprecated in favor of the former, so let's
use the recommended API.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20695
2024-09-19 18:10:07 +03:00
Kefu Chai
1fd1698a90 test: btree: use BOOST_DATA_TEST_CASE() when appropriate
instead grouping tests with different parameters, let's parameterize
them using `BOOST_DATA_TEST_CASE()`, simpler this way. and the tests
can be more structured.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20697
2024-09-19 18:09:05 +03:00
Avi Kivity
6f7c2ce0aa Merge 'cql_server::connection: Process rebounce message in case of multiple shard migrations' from Sergey Zolotukhin
During a query execution, the query can be re-bounced to another shard if the requested data is located there. Previous implementation assumed that the shard cannot be changed after first re-bounce, however with the introduction of Tablets, data could be migrated to another shard after the query was already re-bounced, causing a failure of the query execution. To avoid this issue, the query is re-bounced as needed until it is executed on the correct shard.

Fixes #15465

Closes scylladb/scylladb#20493

* github.com:scylladb/scylladb:
  cql_server: Add a test for multiple query msg rebounces.
  cql_server::connection: process: rebounce msg if needed
  cql_server::connection: process: co-routinize connection::process_on_shard
  cql_server: connection: process: fixup indentation
  cql_server: connection: process_on_shard: drop permit parameter
  transport: server: pass bounce_to_shard as foreign shared ptr
  cql_server: connection: process: add template concept for process_fn
  cql_server: move process_fn_return_type to class definition
2024-09-19 17:27:55 +03:00
Benny Halevy
574a08ed96 storage_service: rebuild: warn about tablets-enabled keyspaces
Until we automatically support rebuild for tablets-enabled
keyspaces, warn the user about them.

The reason this is not an error, is that after
increasing RF in a new datacenter, the current procedure
is to run `nodetool rebuild` on all nodes in that dc
to rebuild the new vnode replicas.
This is not required for tablets, since the additional
replicas are rebuilt automatically as part of ALTER KS.

However, `nodetool rebuild` is also run after local
data loss (e.g. due to corruption and removal of sstables).
In this case, rebuild is not supported for tablets-enabled
keyspaces, as tablet replicas that had lost data may have
already been migrated to other nodes, and rebuilding the
requested node will not know about it.
It is advised to repair all nodes in the datacenter instead.

Refs scylladb/scylladb#17575

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes scylladb/scylladb#20375
2024-09-19 14:25:46 +03:00
Anna Stuchlik
cdc69b4e06 doc: enable publishing docs for branch-6.2
This commit enables publishing documentation from branch-6.2. The docs will be published as UNSTABLE (the warning about version 6.1 being unstable will be displayed).

Fixes https://github.com/scylladb/scylladb/issues/20643

No backport is required.

Closes scylladb/scylladb#20647
2024-09-19 09:39:58 +03:00
Anna Stuchlik
400a14eefa doc: update the unified installer instructions
This commit updates the unified installer instructions to avoid specifying a given version.
At the moment, we're technically unable to use variables in URLs, so we need to update
the page each release.

Fixes https://github.com/scylladb/scylladb/issues/20677

Closes scylladb/scylladb#20680
2024-09-19 09:28:44 +03:00
Anna Stuchlik
aa0c95c95c doc: fix a broken link
This commit fixes a link to the Manager by adding a missing underscore
to the external link.

Closes scylladb/scylladb#20656
2024-09-19 09:20:20 +03:00
Calle Wilund
60f8a9f39d database: Also forced new schema commitlog segment on user initiated memtable flush
Refs #20686
Refs #15607

In #15060 we added forced new commitlog segment on user initated flush,
mainly so that tests can verify tombstone gc and other compaction related
things, without having to wait for "organic" segment deletion.
Schema commitlog was not included, mainly because we did not have tests
featuring compaction checks of schema related tables, but also because
it was assumed to be lower general througput.
There is however no real reason to not include it, and it will make some
testing much quicker and more predictable.

Closes scylladb/scylladb#20691
2024-09-19 09:00:33 +03:00
Benny Halevy
5ccdf1cf1c gms: inet_address: modernize comparison operators
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-09-18 17:07:51 +03:00
Benny Halevy
38540d89a1 gms: inet_address: drop unused raw_addr method
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-09-18 14:21:18 +03:00
Kefu Chai
b0696bd842 test: btree: use BOOST_DATA_TEST_CASE to structure parameterized tests
for better readability. and for more structured tests.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20516
2024-09-18 14:16:28 +03:00
Pavel Emelyanov
eb22c2a8c8 Merge 'reader_concurrency_semaphore: improve the diagnostics dump' from Botond Dénes
* Also dump diagnostics when a read times out while active (not queued).
* Add the "Trigger permit" line, containing the details of the permit which caused the diagnostics dump (by e.g. timing out).
* Add the "Identified bottleneck(s)" line, containing the identified bottlenecks which lead to permits being queued. This line is missing if no such bottleneck can be identified.
* Document the new features, as well as the stat dump, which was added some time ago.

Example of the new dump format:
```
INFO  2024-09-12 08:09:48,046 [shard  0:main] reader_concurrency_semaphore - Semaphore reader_concurrency_semaphore_dump_reader_diganostics with 8/10 count and 106192275/32768 memory resources: timed out, dumping permit diagnostics:
Trigger permit: count=0, memory=0, table=ks.tbl0, operation=mutation-query, state=waiting_for_admission
Identified bottleneck(s): memory

permits count   memory  table/operation/state
3       2       26M     *.*/push-view-updates-2/active
3       2       16M     ks.tbl1/push-view-updates-1/active
1       1       15M     ks.tbl2/push-view-updates-1/active
1       0       13M     ks.tbl1/multishard-mutation-query/active
1       0       12M     ks.tbl0/push-view-updates-1/active
1       1       10M     ks.tbl3/push-view-updates-2/active
1       1       6060K   ks.tbl3/multishard-mutation-query/active
2       1       1930K   ks.tbl0/push-view-updates-2/active
1       0       1216K   ks.tbl0/multishard-mutation-query/active
6       0       0B      ks.tbl1/shard-reader/waiting_for_admission
3       0       0B      *.*/data-query/waiting_for_admission
9       0       0B      ks.tbl0/mutation-query/waiting_for_admission
2       0       0B      ks.tbl2/shard-reader/waiting_for_admission
4       0       0B      ks.tbl0/shard-reader/waiting_for_admission
9       0       0B      ks.tbl0/data-query/waiting_for_admission
7       0       0B      ks.tbl3/mutation-query/waiting_for_admission
5       0       0B      ks.tbl1/mutation-query/waiting_for_admission
2       0       0B      ks.tbl2/mutation-query/waiting_for_admission
8       0       0B      ks.tbl1/data-query/waiting_for_admission
1       0       0B      *.*/mutation-query/waiting_for_admission
26      0       0B      permits omitted for brevity

96      8       101M    total

Stats:
permit_based_evictions: 0
time_based_evictions: 0
inactive_reads: 0
total_successful_reads: 0
total_failed_reads: 0
total_reads_shed_due_to_overload: 0
total_reads_killed_due_to_kill_limit: 0
reads_admitted: 1
reads_enqueued_for_admission: 82
reads_enqueued_for_memory: 0
reads_admitted_immediately: 1
reads_queued_because_ready_list: 0
reads_queued_because_need_cpu_permits: 82
reads_queued_because_memory_resources: 0
reads_queued_because_count_resources: 0
reads_queued_with_eviction: 0
total_permits: 97
current_permits: 96
need_cpu_permits: 0
awaits_permits: 0
disk_reads: 0
sstables_read: 0
```

Fixes: https://github.com/scylladb/scylladb/issues/19535

Improvement, no backport needed.

Closes scylladb/scylladb#20545

* github.com:scylladb/scylladb:
  docs/dev/reader-concurrency-semaphore.md: update the documentation on diagnostics dumps
  test/boost/reader_concurrency_semaphore_test: test the new diagnostics functionality
  reader_concurrency_semaphore: add bottleneck self-diagnosis to diagnosis dump
  reader_concurrency_semaphore: include trigger permit in diagnostic dump
  reader_concurrency_semaphore: propagate permit to do_dump_reader_permit_diagnostics()
  reader_concurrency_semaphore: use consistent exception type for timeout
  reader_concurrency_semaphore: dump diagnostics when non-waiting reader times out
2024-09-18 14:06:05 +03:00
Botond Dénes
1efda557b1 replica/table: query_mutations(): enter the table's async gate
So the table is not dropped while the query is ongoing.
query() already does this but using old-fashioned enter()+leave(),
convert it to use the new RAII helper.

Closes scylladb/scylladb#20583
2024-09-18 14:03:22 +03:00
Pavel Emelyanov
2f4f0eb060 Merge 'Alternator: a few RBAC fixes' from Nadav Har'El
The main goal of this PR is to fix a bug (#20619) in the alternator_enforce_authorization=false setting - which didn't do its job (i.e, _don't_ check permissions) when authorization is configured in CQL but not wanted in Alternator.

The series also a few smaller bugs in the code that were discovered while debugging the main issue:
1. A potential use-after-free (that didn't seem to hit us in practice) is fixed.
2. A confusing error message (that was also reported in #20619) is improved.
3. Make the alternator_enforce_authorization live-updatable. There was no reason why it shouldn't be, and as this series needs to make this flag available to more code, let's just do it properly and assume the flag is live-updatable.

Because the RBAC feature has not been backported to any open-source branches, neither should these fixes. But if some private branch received a backport of the RBAC feature, it should get these fixes too.

Fixes #20619.

Closes scylladb/scylladb#20640

* github.com:scylladb/scylladb:
  alternator: make alternator_enforce_authorization live-updateable
  alternator: fix alternator_enforce_authorization=false
  alternator: improve error message when unauthenticated
  alternator: avoid use-after-free in RBAC
2024-09-18 14:02:09 +03:00
Kefu Chai
cb1670b79b Update seastar submodule
* seastar ec5da7a6...69f88e2f (38):
  > build: s/Sanitizers_COMPILER_OPTIONS/Sanitizers_COMPILE_OPTIONS
  > test: Update httpd test with request/reply body writing sugar
  > http: Add sugar to request and response body writers
  > utils: Add util::write_to_stream() helper
  > seastar-addr2line: adjust llvm termination regex
  > README.md: add Crimson project
  > rpc: conditionally use fmt::runtime() based on SEASTAR_LOGGER_COMPILE_TIME_FMT
  > build: check the combination of Sanitizers
  > tls: clear session ticket before releasing
  > print: remove dead code
  > doc/lambda-coroutine-fiasco: reword for better readability
  > rpc: fix compilation error caused by fmt::runtime()
  > tutorial: explain the use case of rethrow_exception and coroutine::exception
  > reactor: print more informative error when io_submit fails
  > README.md: note GitHub discussions
  > prometheus: `fmt::print` to stringstream directly
  > doc: add document for testing with seastar
  > seastar/testing: only include used headers
  > test: Add abortable http client test cases
  > http/client: Add abortable make_request() API method
  > http/client: Abort established connections
  > http/client: Handle abort source in pool wait
  > http/client: Add abort source to factory::make() method
  > http/client: Pass abort_source here and there
  > http/client: Idnentation fix after previous patch
  > http/client: Merge some continuations explicitly
  > signal: add seastar signal api
  > httpd: remove unused prometheus structs
  > print: use fmtlib's fmt::format_string in format()
  > rpc: do not use seastar::format() in rpc logger
  > treewide: s/format/seastar::format/
  > prometheus: sanitize label value for text protocol
  > tests: unit test prometheus wire format
  > io-tester: Introduce batches to rate-based submission
  > io-tester: Generalize issueing request and collecting its result
  > io-tester: Cancel intent once
  > io-tester: Dont carry rps/parallelism variables over lambdas
  > io-tester: Simplify in-flight management

The breaking changes in the seastar submodule necessitate corresponding
modifications in our code. These changes must be implemented together in
a single commit to maintain consistency. So that each commit is buildable.

following changes are included in addition to seastar submodule update:
* instead of passing a `const char*` for the format string, pass a
  templated `fmt::format_string<...>`, this depends on the
  `seastar::format()` change in seastar.
* explicitly call `fmt::runtime()` if the format string is not a
  consteval expression. this depends on the `seastar::format()` change
  in seastar. as `seastar::format()` does not accept a plain
  `const char*` which is not constexpr anymore.
* pass abort_source to `dns_connection_factory::make()`. this depends on
  the change in seastar, which added a `abort_source*` argument to
  the pure virtual member function of `connection_factory::make()`.
* call call {fmt,seastar}::format() explicitly. this is a follow up of
  3e84d43f, which takes care of all places where we should call
  `fmt::format()` and `seastar::format()` explicitly to disambiguate the
  `format()` call. but more `format()` call made their way into the source
   tree after 3e84d43f. so we need fix them as well.
* include used header in tests

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Update seastar submodule

 Please enter the commit message for your changes. Lines starting

Closes scylladb/scylladb#20649
2024-09-18 13:59:22 +03:00
Anna Mikhlin
0c7ca284ad mergify: add support for branch-6.2
branch-6.2 is already available, adding support for it in mergify to
allow backport to this new branch.
in addition, since branch 5.4 reached EOL - removing it

Closes scylladb/scylladb#20669
2024-09-18 08:30:41 +03:00
Ernest Zaslavsky
924325fd25 treewide: add "prefix" parameter to backup API
Allow the caller to pass the prefix when performing backup and restore

Fixes scylladb/scylladb#20335

Closes scylladb/scylladb#20413
2024-09-18 08:25:00 +03:00
Calle Wilund
b789361091 commitlog: Fix assertion in oversized_alloc
Fixes #20633

Cannot assert on actual request_controller when releasing permit, as the
release, if we have waiters in queue, will subtract some units to hand to them.
Instead assert on permit size + waiter status (and if zero, also controller value)

* v2 - use SCYLLA_ASSERT

Closes scylladb/scylladb#20654
2024-09-18 08:22:28 +03:00
Avi Kivity
57ab5ce313 repair: row_level: simplify repair_put_row_diff_with_rpc_stream_process_op()
repair_put_row_diff_with_rpc_stream_process_op() always returns
stop_iteration::no (or throws). Moreover, the return value is ignored
by its only caller. Simplify by returning a plain future<>.

Closes scylladb/scylladb#20610
2024-09-18 08:17:09 +03:00
Botond Dénes
d72fcb11f5 Merge 'Add new GDB commands to dump sstable index file from memory and print promoted index ' from Tomasz Grabiec
Closes scylladb/scylladb#20648

* github.com:scylladb/scylladb:
  gdb: Introduce "scylla sstable-dump-cached-index" command
  gdb: Introduce "scylla sstable-promoted-index" command
  gdb: Fix range printer for singular ranges
2024-09-18 08:13:04 +03:00
Nadav Har'El
24fb92c8ba Merge 'cql3: simplify runtime component of selection filtering' from Avi Kivity
Most of the analysis of the WHERE clause is done in statement_restrictions. It determines
what parts to use for the primary or secondary index, and what parts to use for filtering.

The difficult part is that it has a very wide interface. After construction, the user must pick
the correct bits from many public functions. There are subtle interactions between them
that are hard to untangle.

This series simplifies the interface as it is used for selection filtering. In the end, only
two public functions are used, both returning expressions: one for the partition-level
filtering, one for the clustering row level filtering.

In the end, the WHERE clause is factored into three parts:
 - one part goes into the read_command of the primary or secondary index
 - another part (that references only partition key columns and static key columns) is used to filter entire partitions
 - another part (that currently references only clustering key columns and regular columns, but one day may reference other columns) is used to filter clustering rows

Refactoring, no backport.

Closes scylladb/scylladb#20487

* github.com:scylladb/scylladb:
  cql3: statement_restrictions: drop accessors for single-column key restrictions
  cql3: selection: adjust indentation
  cql3: selection: delete empty loop
  cql3: statement_restrictions, selection: fold multi-column restrictions into row-level filter
  cql3: statement_restrictions, selection: merge clustering key filter and regular columns filter
  cql3: statement_restrictions, selection: merge partition key filter and static columns filter
  cql3: selection: filter regular and static rows as a single expression each
  cql3: statement_restrictions: collect regular column and static column filters into single expressions
  cql3: selection: filter clustering key as a single expression
  cql3: statement_restrictions: expose filter for clustering key
  cql3: selection: filter partition key as a single expression
  cql3: statement_restrictions: expose filter for partition key
  cql3: statement_restrictions: remove relations used for indexing from filtering
  cql3: statement_restrictions: bail out of find_idx if !_uses_secondary_index
  cql3: statement_restrictions, modification_statement: pass correct value of check_indexes
  cql3: statement_restrictions: correct mismatched clustering/partition restrictions references
  cql3: statement_restrictions: precalculate get_column_defs_for_filtering()
  cql3: selection: do_filter(): push static/regular row glue to higher level
2024-09-17 22:58:24 +03:00
Piotr Dulikowski
cc5c3aaae7 Merge 'message/messaging_service: guard adding maintenance tenant under cluster feature' from Michał Jadwiszczak
In https://github.com/scylladb/scylladb/pull/18729, we introduced a new statement tenant `$maintenance`, but the change wasn't protected by any cluster feature.
This wasn't a problem for OSS, since unknown isolation cookie just uses default scheduling group. However, in enterprise that leads to creating a service level on not-upgraded nodes, which may end up in an error if user create maximum number of service levels.

This patch adds a cluster feature to guard adding the new tenant. It's done in the way to handle two upgrade scenarios:
- version without `$maintenance` tenant -> version with `$maintenance` tenant guarded by a feature
- version with `$maintenance` tenant but not guarded by a feature -> version with `$maintenance` tenant guarded by a feature

The PR adds `enabled` flag to statement tenants.
This way, when the tenant is disabled, it cannot be used to create a connection, but it can be used to accept an incoming connection.
The `$maintenance` tenant is added to the config as disabled and it gets enabled once the corresponding feature is enabled.

Fixes scylladb/scylladb#20070
Refs scylladb/scylla-enterprise#4403

Closes scylladb/scylladb#19802

* github.com:scylladb/scylladb:
  message/messaging_service: guard adding maintenance tenant under cluster feature
  message/messaging_service: add feature_service dependency
  message/messaging_service: add `enabled` flag to statement tenants
2024-09-17 18:24:34 +02:00
Avi Kivity
1663fbe717 cql3: statement_restrictions: use functional style
Instead of a constructor, use a new function
analyze_statement_restrictions() as the entry point. It returns an
immutable statement_restrictions object.

This opens the door to returning a variant, with each arm of the variant
corresponding to a different query plan.
2024-09-17 17:13:27 +03:00
Avi Kivity
3169b8e0ec cql3: statement_restrictions: calculate the index only once
find_idx() is called several times. Rename it do_find_idx(), call it
just once, store the results, and make find_idx() return the stored
results.

This simplifies control flow and reduces the risk that successive
calls of find_idx return different results.
2024-09-17 17:03:31 +03:00
Avi Kivity
d5c8083b76 cql3: statement_restrictions: make it a const object
Make validate_secondary_index_selections() const (it trivially is),
and call prepare_indexed_local() / prepared_indexed_global() at the
end of the constructor.

By making statement_restrictions a const object, reasoning about it
can be local (looking at the source file) rather than global (looking
at all the interactions of the class with its environment. In fact,
we might make it a function one day.

Since prepare_indexed_global()/prepare_indexed_local() only mutate
_idx_tbl_ck_prefix, which isn't mutated by the rest of the code, the
transformation is safe.

The corresponding code is removed from select_statement. The removal
isn't complete since it still uses some computation, but later
deduplication is left for another day.
2024-09-17 17:03:27 +03:00
Sergey Zolotukhin
68740f57c2 cql_server: Add a test for multiple query msg rebounces.
The test emulates several LWT(Lightweight Transaction) query rebounces. Currently, the code
that processes queries does not expect that a query may be rebounced more than once.
It was impossible with the VNodes, but with intruduction of the Tablets, data can be moved
between shards by the balancer thus a query can be rebounced to different shards multiple times.
2024-09-17 15:19:56 +02:00
Benny Halevy
65430b9e1b cql_server::connection: process: rebounce msg if needed
Rebounce the msg to another shard if needed,
e.g. in the case of tablet migration.

An example for that, as given by Tomasz Grabiec:
> Bouncing happens when executing LWT statement in
> modification_statement::execute_with_condition by returning a
> special result message kind. The code assumes that after
> jumping to the shard from the bounce request, the result
> message is the regular one and not yet another bounce.
> There is no problem with vnodes, because shards don't change.
> With tablets, they can change at run time on migration.

Fixes scylladb/scylladb#15465

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-09-17 15:09:43 +02:00
Sergey Zolotukhin
f674f522aa cql_server::connection: process: co-routinize connection::process_on_shard
`cql_server::connection::process_on_shard` is made a co-routine to
make sure captured objects' lifetime is managed by the source shard,
avoiding error prone inter-shard objects transfers.
2024-09-17 14:54:42 +02:00
Nadav Har'El
17deaae463 alternator: make alternator_enforce_authorization live-updateable
For no good reason, the "alternator_enforce_authorization" flag (which
chooses whether to enable authentication and authorization checks in
Alternator) was not live-updatable, so make it so.

Both "server" and "executor" objects use this configuration flag, the
former is fixed in this patch (to hold a live-updatable reference
instead of a copy of a boolean), the latter was already prepared for
this change and already held a live-updatable reference.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2024-09-17 15:51:16 +03:00
Nadav Har'El
00793059e1 alternator: fix alternator_enforce_authorization=false
When the configuration has alternator_enforce_authorization=false,
Alternator should not do authentication (check which user signed each
request) nor authorization (check if that user has permissions to do
each operation).

Our implementation forgot to disable the authorization checks when
it's configured to false. The (incorrect) assumption was that when
alternator_enforce_authorization is configured to false, the CQL
'authenticator' and 'authorizer' configuration is also disabled -
so the authorization checks will be no-ops. But we can't assume
that: Users are free to configure 'authenticator' and 'authorizer'
for use in CQL, and then set alternator_enforce_authorization=false
just for Alternator.

So this patch adds a new test for this case - when we have
authenticator=PasswordAuthenticator, authorizer=CassandraAuthorizer
but alternator_enforce_authorization=false, and fixes it to work
correctly.

The heart of the fix is trivial: the `verify_*_permission()` functions
just need to check the alternator_enforce_authorization and return
immediately when false. The bigger part of this change is to get the
alternator_enforce_authorization into the "executor" object and then
to pass it into the verify calls.

Although alternator_enforce_authorization is not YET live updatable,
this code is prepared for the future that it may become live
updatable, so the executor object saves not the boolean value of
this flag, but a live-updatable reference to it.

Fixes #20619

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2024-09-17 15:50:00 +03:00
Nadav Har'El
76af7c0389 alternator: improve error message when unauthenticated
When access-control checks report permission denied, we want to report
the name of the authenticated role (the role signing the request) which
didn't have the permission. When authentication was disabled, and there
is no authenticated role, we printed the fake name "anonymous", but this
can confuse users (it confused me!) to think there's an actual role
named "anonymous". So let's change that string to "<anonymous>" with
angle brackets - it makes it more obvious that this isn't a real role,
but actually an anonymous request.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2024-09-17 15:44:29 +03:00
Tomasz Grabiec
e70ce4d6ed gdb: Introduce "scylla sstable-dump-cached-index" command 2024-09-17 14:41:18 +02:00