Commit Graph

42402 Commits

Author SHA1 Message Date
Yaron Kaikov
2cf7cc1ea5 scylla_setup: Remove jmx and tools packages from being verified
Following
b8634fb244
machine image started to fail with the following error:
```
10:44:59  ␛[0;32m    googlecompute.gce: scylla-jmx package is not installed.␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce: Traceback (most recent call last):␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce:   File "/home/ubuntu/scylla_install_image", line 135, in <module>␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce:     run('/opt/scylladb/scripts/scylla_setup --no-coredump-setup --no-sysconfig-setup --no-raid-setup --no-io-setup --no-ec2-check --no-swap-setup --no-cpuscaling-setup --no-ntp-setup', shell=True, check=True)␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce:   File "/usr/lib/python3.10/subprocess.py", line 526, in run␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce:     raise CalledProcessError(retcode, process.args,␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce: subprocess.CalledProcessError: Command '/opt/scylladb/scripts/scylla_setup --no-coredump-setup --no-sysconfig-setup --no-raid-setup --no-io-setup --no-ec2-check --no-swap-setup --no-cpuscaling-setup --no-ntp-setup' returned non-zero exit status 1.␛[0m
```

It seems we no longer need to verify that jmx and tools-java packages are installed.

Closes scylladb/scylladb#18494
2024-05-02 13:30:50 +03:00
Kefu Chai
f183f5aa80 Update seastar submodule
* seastar 2b43417d...b73e5e7d (11):
  > treewide: inherit from formatter<string_view> not formatter<std::string_view>
  > CMakeLists.txt: Apply CXX deprecated flags conditionally
  > tls: add assignment operator for gnutls_datum
  > tls: s/get0()/get()/
  > io_queue: do not reference moved variable
  > TLS: use helper function in get_distinguished_name & get_alt_name_information
  > TLS: Add support for TLS1.3 session tickets
  > iotune: ignore shards with id above max_iodepth
  > core/future: remove a template parameter from set_callback()
  > util: with_file_input_stream: always close file
  > core/sleep: Use more raii-sh aproach to maintain sleeper

Fixes #5181

Closes scylladb/scylladb#18491
2024-05-02 07:35:42 +03:00
Takuya ASADA
b8634fb244 dist: stop installing scylla-tools, scylla-jmx by default
Since we added native nodetool, we no longer need to install scylla-tools
and scylla-jmx, drop them from scylla metapackage and make it optional
package.

Closes #18472

Closes scylladb/scylladb#18487
2024-05-01 22:15:40 +03:00
Kefu Chai
af5674211d redis/server.hh: suppress -Wimplicit-fallthrough from protocol_parser.hh
when compiling the tree with clang-18 and ragel 6.10, the compiler
warns like:

```
/usr/local/bin/cmake -E __run_co_compile --tidy="clang-tidy-18;--checks=-*,bugprone-use-after-move;--extra-arg-before=--driver-mode=g++" --source=/home/runner/work/scylladb/scylladb/redis/controller.cc -- /usr/bin/clang++-18 -DBOOST_NO_CXX98_FUNCTION_BASE -DSCYLLA_BUILD_MODE=release -DSEASTAR_API_LEVEL=7 -DSEASTAR_LOGGER_COMPILE_TIME_FMT -DSEASTAR_LOGGER_TYPE_STDOUT -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -DSEASTAR_SSTRING -DXXH_PRIVATE_API -I/home/runner/work/scylladb/scylladb -I/home/runner/work/scylladb/scylladb/build/gen -I/home/runner/work/scylladb/scylladb/seastar/include -I/home/runner/work/scylladb/scylladb/build/seastar/gen/include -I/home/runner/work/scylladb/scylladb/build/seastar/gen/src -isystem /home/runner/work/scylladb/scylladb/cooking/include -ffunction-sections -fdata-sections -O3 -g -gz -std=gnu++20 -fvisibility=hidden -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wimplicit-fallthrough -Wno-c++11-narrowing -Wno-deprecated-copy -Wno-mismatched-tags -Wno-missing-field-initializers -Wno-overloaded-virtual -Wno-unsupported-friend -Wno-enum-constexpr-conversion -Wno-unused-parameter -ffile-prefix-map=/home/runner/work/scylladb/scylladb=. -march=westmere -mllvm -inline-threshold=2500 -fno-slp-vectorize -U_FORTIFY_SOURCE -Werror=unused-result -MD -MT redis/CMakeFiles/redis.dir/controller.cc.o -MF redis/CMakeFiles/redis.dir/controller.cc.o.d -o redis/CMakeFiles/redis.dir/controller.cc.o -c /home/runner/work/scylladb/scylladb/redis/controller.cc
error: too many errors emitted, stopping now [clang-diagnostic-error]
Error: /home/runner/work/scylladb/scylladb/build/gen/redis/protocol_parser.hh:110:1: error: unannotated fall-through between switch labels [clang-diagnostic-implicit-fallthrough]
  110 | case 1:
      | ^
/home/runner/work/scylladb/scylladb/build/gen/redis/protocol_parser.hh:110:1: note: insert 'FMT_FALLTHROUGH;' to silence this warning
  110 | case 1:
      | ^
      | FMT_FALLTHROUGH;
```

since we have `-Werror`, the warnings like this are considered as error,
hence the build fails. in order to address this failure, let's silence
this warning when including this generated header file.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18447
2024-05-01 18:47:24 +03:00
Kefu Chai
08d1362f80 utils/chunked_vector: fix some typos in comment
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18486
2024-05-01 16:38:43 +03:00
Nadav Har'El
4e78e2d506 test/cql-pytest, cdc: add test for what happens when log name is taken
In our CDC implementation, the CDC log table for table "xyz" is always
called "xyz_scylla_cdc_log". If this table name is taken, and the user
tries to create a table "xyz" with CDC enabled - or enable CDC on the
table "xyz", the creation/enabling should fail gracefully, with a clear
error message. This test verifies this.

The new test passes - the code is already correct. I just wanted to
verify that it is (and to prevent future regressions).

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes scylladb/scylladb#18485
2024-05-01 14:46:19 +03:00
Pavel Emelyanov
5d992a4f01 proxy: Remove declaration of nonexisting view_update_write_response_handler class
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>

Closes scylladb/scylladb#18417
2024-05-01 10:15:41 +03:00
Botond Dénes
65a385f5d0 Merge 'Relax the way view builder code checks if a table exists' from Pavel Emelyanov
There are two places that workaround db.column_family_exists() call with some fancy exceptions-catching lambda.
This PR makes things simpler.

Closes scylladb/scylladb#18441

* github.com:scylladb/scylladb:
  view: Open-code one line lambda checking if table exists
  view: Use non-throwoing check if a table exists
2024-05-01 10:14:58 +03:00
Kefu Chai
94ac0799d9 build: cmake: link scylla_tracing against scylla-main
because tracing/trace_keyspace_helper.cc references symbols
defined by table_helper, which is in turn provided by scylla-main,
we should link tracing_tracing against scylla-main.

otherwise we could have following link failure:

```
./build/./tracing/trace_keyspace_helper.cc:214: error: undefined reference to 'table_helper::setup_keyspace(cql3::query_processor&, service::migration_manager&, std::basic_string_view<char, std::char_traits<char> >, seastar::basic_sstring<char, unsigned int, 15u, true>, service::query_state&, std::vector<table_helper*, std::allocator<table_helper*> >)'
./build/./tracing/trace_keyspace_helper.cc:396: error: undefined reference to 'table_helper::cache_table_info(cql3::query_processor&, service::migration_manager&, service::query_state&)'
./table_helper.hh:92: error: undefined reference to 'table_helper::insert(cql3::query_processor&, service::migration_manager&, service::query_state&, seastar::noncopyable_function<cql3::query_options ()>)'
./table_helper.hh:92: error: undefined reference to 'table_helper::insert(cql3::query_processor&, service::migration_manager&, service::query_state&, seastar::noncopyable_function<cql3::query_options ()>)'
./table_helper.hh:92: error: undefined reference to 'table_helper::insert(cql3::query_processor&, service::migration_manager&, service::query_state&, seastar::noncopyable_function<cql3::query_options ()>)'
./table_helper.hh:92: error: undefined reference to 'table_helper::insert(cql3::query_processor&, service::migration_manager&, service::query_state&, seastar::noncopyable_function<cql3::query_options ()>)'
clang++-18: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
```

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18455
2024-05-01 10:08:11 +03:00
Kefu Chai
f0d12df7fc reloc: create $BUILDDIR for getting its path
when building with CMake, there is a use case where the $BUILDIR
is not created yet, when `reloc/build_rpm.sh` is launched. in order
to enable us to run this script without creating $BUILDIR first, let's
create this directory first.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18464
2024-05-01 09:52:17 +03:00
Kefu Chai
8168f02550 raft_group_registry: do not use moved variable
clang-tidy warns like:
```
[628/713] Building CXX object service/CMakeFiles/service.dir/raft/raft_group_registry.cc.o
Warning: /home/runner/work/scylladb/scylladb/service/raft/raft_group_registry.cc:543:66: warning: 'id' used after it was moved [bugprone-use-after-move]
  543 |             auto& rate_limit = _rate_limits.try_get_recent_entry(id, std::chrono::minutes(5));
      |                                                                  ^
/home/runner/work/scylladb/scylladb/service/raft/raft_group_registry.cc:539:19: note: move occurred here
  539 |     auto dst_id = raft::server_id{std::move(id)};
      |                   ^
```

this is a false alarm. as the type of `id` is actually `utils::UUID`
which is a struct enclosing two `int64_t` variables. and we don't
define a move constructor for `utils::UUID`. so the value of of `id`
is intact after being moved away. but it is still confusing at
the first glance, as we are indeed referencing a moved-away variable.

so in order to reduce the confusion and to silence the warning, let's
just do not `std::move(id)`.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18449
2024-05-01 09:45:12 +03:00
Kefu Chai
bd0d246b57 tools/scylla-nodetool: implement the resetlocalschema command
Fixes #18468
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18470
2024-05-01 08:49:11 +03:00
Raphael S. Carvalho
b980634ff2 test: Verify tablet cleanup is properly retried on failure
Doesn't test only coordinator ability to retry on failure, but also
that replica will be able to properly continue cleanup of a storage
group from where it left off (when failure happened), not leave any
sstables behind.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>

Closes scylladb/scylladb#18426
2024-04-30 19:27:17 +02:00
Raphael S. Carvalho
62b1cfa89c topology_coordinator: Fix synchronization of tablet split with other concurrent ops
Finalization of tablet split was only synchronizing with migrations, but
that's not enough as we want to make sure that all processes like repair
completes first as they might hold erm and therefore will be working
with a "stale" version of token metadata.

For synchronization to work properly, handling of tablet split finalize
will now take over the state machine, when possible, and execute a
global token metadata barrier to guarantee that update in topology by
split won't cause problems. Repair for example could be writing a
sstable with stale metadata, and therefore, could generate a sstable
that spans multiple tablets. We don't want that to happen, therefore
we need the barrier.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>

Closes scylladb/scylladb#18380
2024-04-30 19:23:28 +02:00
Botond Dénes
525553aa41 SCYLLA-VERSION-GEN: warn against using - or _ in custom version names
Doing so is a pitfall that will make one waste a lot of time rebuilding
the packages, just because at the end it turns out that the version has
illegal characters in it. The author of this patch has certainly fallen
into this pitfall a lot of times.

Closes scylladb/scylladb#18429
2024-04-30 18:14:51 +03:00
Avi Kivity
ea15ddc7dc Merge 'Fix population of non-normal sstables from registry' from Pavel Emelyanov
On boot sstables are populated from normal location as well as from quarantine and staging. It turned out that sstables listed in registry (S3-backed ones) are not populated from non-normal states.

Closes scylladb/scylladb#18439

* github.com:scylladb/scylladb:
  test: Add test for how quarantined sstables registry entries are loaded
  sstable_directory: Use sstable location to initialize registry lister
2024-04-30 18:10:11 +03:00
Avi Kivity
329b135b5e Merge 'chunked_vector: fix use after free in emplace back' from Benny Halevy
Currently, push_back or emplace_back reallocate the last chunk
before constructing the new element.

If the arg passed to push_back/emplace_back is a reference to an
existing element in the vector, reallocating the last chunk will
invalidate the arg reference before it is used.

This patch changes the order when reallocating
the last chunk in reserve_for_emplace_back:
First, a new chunk_ptr is allocated.
Then, the back_element is emplaced in the
newly allocated array.
And only then, existing elements in the current
last chunk are migrated to the new chunk.
Eventually, the new chunk replaces the existing chunk.

If no reservation is requried, the back element
is emplaced "in place" in the current last chunk.

Fixes scylladb/scylladb#18072

Closes scylladb/scylladb#18073

* github.com:scylladb/scylladb:
  test: chunked_managed_vector_test: add test_push_back_using_existing_element
  utils: chunked_vector: reserve_for_emplace_back: emplace before migrating existing elements
  utils: chunked_vector: push_back: call emplace_back
  utils: chunked_vector: define min_chunk_capacity
  utils: chunked*vector: use std::clamp
2024-04-30 18:09:04 +03:00
David Garcia
f62197ee1e docs: enable concurrent downloads
Downloads chunks of 10 CSV concurrently to speed up doc builds.

Closes scylladb/scylladb#18469
2024-04-30 16:13:40 +03:00
Raphael S. Carvalho
d7a01598ce tools: Make sstable shard-of efficient by loading minimum to compute owners
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>

Closes scylladb/scylladb#18440
2024-04-30 16:10:58 +03:00
Kefu Chai
3a1ceb96d7 utils: UUID_gen: include <atomic>
in UUID_gen.cc, we are using `std::atomic<int64_t>` in
`make_thread_local_node()`, but this template is not defined by
any of the included headers. but  we should include used headers
to be self-contained.

when compiling on ubuntu:jammy with libstdc++-13, we have following
error:
```
/usr/local/bin/cmake -E __run_co_compile --tidy="clang-tidy-18;--checks=-*,bugprone-use-after-move;--extra-arg-before=--driver-mode=g++" --source=/home/runner/work/scylladb/scylladb/utils/UUID_gen.cc -- /usr/bin/clang++-18 -DBOOST_ALL_NO_LIB -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_REGEX_DYN_LINK -DSCYLLA_BUILD_MODE=release -DSEASTAR_API_LEVEL=7 -DSEASTAR_LOGGER_COMPILE_TIME_FMT -DSEASTAR_LOGGER_TYPE_STDOUT -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -DSEASTAR_SSTRING -DXXH_PRIVATE_API -I/home/runner/work/scylladb/scylladb -I/home/runner/work/scylladb/scylladb/seastar/include -I/home/runner/work/scylladb/scylladb/build/seastar/gen/include -I/home/runner/work/scylladb/scylladb/build/seastar/gen/src -isystem /home/runner/work/scylladb/scylladb/cooking/include -ffunction-sections -fdata-sections -O3 -g -gz -std=gnu++20 -fvisibility=hidden -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wimplicit-fallthrough -Wno-c++11-narrowing -Wno-deprecated-copy -Wno-mismatched-tags -Wno-missing-field-initializers -Wno-overl
Error: /home/runner/work/scylladb/scylladb/utils/UUID_gen.cc:29:33: error: implicit instantiation of undefined template 'std::atomic<long>' [clang-diagnostic-error]
   29 |     static std::atomic<int64_t> thread_id_counter;
      |                                 ^
/usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_atomic.h:361:11: note: template is declared here
  361 |     class atomic;
      |           ^
```
so, in this change, we include `<atomic>` to address this
build failure.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18387
2024-04-30 09:07:22 +03:00
Kefu Chai
6a73c911e3 tools: lua_sstable_consumer.cc: be compatible with Lua 5.3's lua_resume()
in Lua 5.3, lua_resume() only accepts three parameters, while in Lua 5.4,
this function accepts four parameters. so in order to be compatible with
Lua 5.3, we should not pass the 4th parameter to this function.
a macro is defined to conditionally pass this parameter based on the
Lua's version.

see https://www.lua.org/manual/5.3/manual.html#lua_resume

Refs 5b5b8b3264
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18450
2024-04-30 09:06:25 +03:00
Piotr Dulikowski
35f456c483 Merge 'Extend ALTER TABLE ... DROP to allow specifying timestamp of column drop' from Michał Jadwiszczak
In order to correctly restore schema from `DESC SCHEMA WITH INTERNALS`, we need a way to drop a column with a timestamp in the past.

Example:
- table t(a int pk, b int)
- insert some data1
- drop column b
- add column b int
- insert some data2

If the sstables weren't compacted, after restoring the schema from description:
- we will loss column b in data2 if we simply do `ALTER TABLE t DROP b` and `ALTER TABLE t ADD b int`
- we will resurrect column b in data1 if we skip dropping and re-adding the column

Test for this: https://github.com/scylladb/scylla-dtest/pull/4122

Fixes #16482

Closes scylladb/scylladb#18115

* github.com:scylladb/scylladb:
  docs/cql: update ALTER TABLE docs
  test/cqlpytest: add test for prepared `ALTER TABLE ... DROP ... USING TIMESTAMP ?`
  test/cql-pytest: remove `xfail` from alter table with timestamp tests
  cql3/statements: extend `ALTER TABLE ... DROP` to allow specifying timestamp of column drop
  cql3/statements: pass `query_options` to `prepare_schema_mutations()`
  cql3/statements: add bound terms to alter table statement
  cql3/statements: split alter_table_statement into raw and prepared
  schema: allow to specify timestamp of dropped column
2024-04-29 14:05:05 +02:00
Takuya ASADA
af0c0ee8af configure.py: revert changing builddir as absolute path
On be3776ec2a, we changed outdir to
absolute path.
This causes "unknown target" error when we build Scylla using the relative
path something like "ninja build/dev/scylla", since the target name
become absolte path.
Revert the change to able to build with the relative path.

Also, change optimized_clang.sh to use relative path for --builddir,
since we reference "../../$builddir/SCYLLA-*-FILE" when we build
submodule, it won't work with absolute path.

Fixes #18321

Closes scylladb/scylladb#18338
2024-04-29 09:35:21 +03:00
Kefu Chai
4433d2e10e build: cmake: let iotune depends on config specific file
before this change, in order to build `${iotune_path}`, we use
the rule to build `app_iotune` but this target is built using
the default build type, see
https://cmake.org/cmake/help/latest/variable/CMAKE_DEFAULT_BUILD_TYPE.html#variable:CMAKE_DEFAULT_BUILD_TYPE
so, if we want to build `${iotune_path}` for the configuration
which is not listed as the first item in `CMAKE_CONFIGURATION_TYPES`,
we would end up with copying an nonexistent file.

to address this issue, we override the this behavior using
the `$<OUTPUT_CONFIG:...>` generator-expression. so that we
can depend on non-unique path. and the file-level dependency
between ${iotune_path} and $<CONFIG>/iotune can be established.

see also
https://cmake.org/cmake/help/latest/generator/Ninja%20Multi-Config.html#custom-commands

Refs #2717

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18395
2024-04-29 09:06:39 +03:00
Kefu Chai
f03f69ad4f partition_version: move the base class in move ctor
before this change, `partition_version` uses a hand-crafted move
constructor. but it suffers from the warning from clang-tidy, which
believe there is a use-after-move issue, as the inner instance of
it's parent class is constructed using
`anchorless_list_base_hook(std::move(pv))`, and its other member
variables are initialized like `_partition(std::move(pv._partition))`

`std::move(pv)` does not do anything, but *indicates* `pv` maybe
moved from. and what is moved away is but the part belong to its
parent class. so this issue is benign.

but, it's still annoying. as we need to tell the genuine issues
reported by clang-tidy from the false alarms. so we have at least
two options:

- stop using clang-tidy
- ignore this warning
- silence this warning using LINT direction in a comment
- use another way to implement the move constructor

in this change, we just cast the moved instance to its
base class and move it instead, this should applease
clang-tidy.

Fixes #18354
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18359
2024-04-28 18:34:45 +02:00
Pavel Emelyanov
7f2742893e view: Open-code one line lambda checking if table exists
Continuation of the previous patch. The lambda in question used to be a
heavyweight(y) code, but now it's one-liner. And it's only called once,
so no more point in keeping it.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2024-04-26 20:19:38 +03:00
Pavel Emelyanov
a3e76f9c93 view: Use non-throwoing check if a table exists
Two places in view code check if a table exists by finding its schema ID
and catching no_such_column_family exception. That's a bit heavyweight,
database has column_family_exists() method for such cases.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2024-04-26 20:17:35 +03:00
Pavel Emelyanov
5e23493d25 test: Add test for how quarantined sstables registry entries are loaded
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2024-04-26 16:54:43 +03:00
Pavel Emelyanov
ba512c52a5 sstable_directory: Use sstable location to initialize registry lister
When populating sstables on boot a bunch of sstable_directory objects is
created. For each sstable there come three -- one for normal, quarantine
and staging state. Each is initialized with sstable location (which is
now a datadir/ks_name/cf_name-and-uuid) and the desired state (a enum
class). When created, the directory object wires up component lister,
depending on which storage options are provided. For local sstables a
legacy filesystem lister is created and it's initialized with a path
where to search files for -- location + / + string(state). But for s3
sstables, that keep their entries in registry, the lister is
errorneously initialized with the same location + / + string(state)
value. The mistake is that sstables in registry keep location and state
in different columns, so for any state lister should query registry with
the same location value (then it filters entries by state on its own).

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2024-04-26 16:36:47 +03:00
Kamil Braun
d8313dda43 Merge 'db: config: move consistent-topology-changes out of experimental and make it the default for new clusters' from Patryk Jędrzejczak
We move consistent cluster management out of experimental and
make it the default for new clusters in 6.0. In code, we make the
`consistent-topology-changes` flag unused and assumed to be true.

In 6.0, the topology upgrade procedure will be manual and
voluntary, so some clusters will still be using the gossip-based
topology even though they support the raft-based topology.
Therefore, we need to continue testing the gossip-based topology.
This is possible by using the `force-gossip-topology-changes` flag
introduced in scylladb/scylladb#18284.

Ref scylladb/scylladb#17802

Closes scylladb/scylladb#18285

* github.com:scylladb/scylladb:
  docs: raft.rst: update after removing consistent-topology-changes
  treewide: fix indentation after the previous patch
  db: config: make consistent-topology-changes unused
  test: lib: single_node_cql_env: restart a node in noninitial run_in_thread calls
  test: test_read_required_hosts: run with force-gossip-topology-changes
  storage_service: join_cluster: replace force_gossip_based_join with force-gossip-topology-changes
  storage_service: join_token_ring: fix finish_setup_after_join calls
2024-04-26 14:45:29 +02:00
Botond Dénes
b96f28356a Merge 'api/storage_service: convert runtime_error from repair to http error ' from Kefu Chai
in `set_repair()`, despite that the repair is performed asynchronously,
we check the options specified by client immediately, and throw
`std::runtime_error`, if any of them is not supported.

before this change, these unhandled exceptions are translated to HTTP
500 error but the underlying HTTP router. but this is misleading, as
these errors are caused by client, not server.

in this change, we handle the `runtime_error`, and translate them
into `httpd::bad_param_exception`, so that the client can have
HTTP 400 (Bad Request) instead of HTTP 500 (Internal Server Error),
and with informative error message.

for instance, if we apply repair with "small_table_optimization" enabled
on a keyspace with tablets enabled. we should have an HTTP error 400
with "The small_table_optimization option is not supported for tablet repair"
as the body of the error. this would much more helpful.

Closes scylladb/scylladb#18389

* github.com:scylladb/scylladb:
  api/storage_service: convert runtime_error from repair to http error
  repair: change runtime_error to invalid_argument in do_repair_start()
  api/storage_service: coroutinize set_repair()
2024-04-26 13:27:51 +03:00
Patryk Jędrzejczak
3a100cd16c test: test_raft_recovery_stuck: ensure raft upgrade procedure failed
We have log browsing in test.py now, so we can fix this TODO easily.

Closes scylladb/scylladb#18425
2024-04-26 10:16:49 +02:00
Asias He
62a9ecff51 repair: Cleanup repair history status entry for tablet
The entry in the repair history map that is used to track repair status
internally for each repair job should be removed after the repair job is
done. We do the same for vnode repairs.

This patch adds the missing automatic history cleanup code which is
missed in the initial tablet repair support in commit 54239514af,
which does not support repair history update back then.

Refs #17046

Closes scylladb/scylladb#18434
2024-04-26 10:56:45 +03:00
Botond Dénes
044fd7a3ec Merge 'Move some view updating methods from table to view_update_generator' from Pavel Emelyanov
The populate_views() and generate_and_propagate_view_updates() both naturally belong to view_update_generator -- they don't need anything special from table itself, but rather depend on some internals of the v.u.generator itself.

Moving them there lets removing the view concurrency semaphore from keyspace and table, thus reducing the cross-components dependencies.

Closes scylladb/scylladb#18421

* github.com:scylladb/scylladb:
  replica: Do not carry view concurrency semaphore pointer around
  view: Get concurrency semaphore via database, not table
  view_update_generator: Mark mutate_MV() private
  view: Move view_update_generator methods' code
  view: Move table::generate_and_propagate_view_updates into view code
  view: Move table::populate_views() into view_update_generator class
2024-04-26 10:55:38 +03:00
Botond Dénes
d566eec89a Merge 'treewide: remove {dclocal_,}read_repair_chance options' from Kefu Chai
dclocal_read_repair_chance and read_repair_chance have been removed in Cassandra 3.11 and 4.x, see
https://issues.apache.org/jira/browse/CASSANDRA-13910. if we expose these properties via DDL, Cassandra would fail to consume the CQL statement creating the table when performing migration from Scylla to Cassandra 4.x, as the latter does not understand these properties anymore.

currently the default values of `dc_local_read_repair_chance` and `read_repair_chance` are both "0". so they are practically disabled, unless user deliberately set them to a value greater than 0.

also, as a side effect, Cassandra 4.x has better support of Python3. the cqlsh shipped along with Cassandra 3.11.16 only supports python2.7, see
https://github.com/apache/cassandra/blob/cassandra-3.11.16/bin/cqlsh.py it errors out if the system only provides python3 with the error of
```
No appropriate python interpreter found.
```
but modern linux systems do not provide python2 anymore.

so, in this change, we deprecate these two options.

Fixes #3502
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18087

* github.com:scylladb/scylladb:
  docs: drop documents related to {,dclocal_}read_repair_chance
  treewide: remove {dclocal_,}read_repair_chance options
2024-04-26 10:48:47 +03:00
Michał Chojnowski
c1146314a1 docs: clarify that DELETE can be used with USING TIMEOUT
The current text seems to suggest that `USING TIMEOUT` doesn't work with `DELETE` and `BATCH`. But that's wrong.

Closes scylladb/scylladb#18424
2024-04-26 10:48:17 +03:00
Pavel Emelyanov
4ac30e5337 view-builder: Print correct exception in built ste exception handler
Inside .handle_exception() continuation std::current_exception() doesn't
work, there's std::exception ex argument to handler's lambda instead

fixes #18423

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>

Closes scylladb/scylladb#18349
2024-04-26 09:58:45 +03:00
Kefu Chai
0bbaded4ce api/storage_service: convert runtime_error from repair to http error
in `set_repair()`, despite that the repair is performed asynchronously,
we check the options specified by client immediately, and throw
`std::runtime_error`, if any of them is not supported.

before this change, these unhandled exceptions are translated to HTTP
500 error but the underlying HTTP router. but this is misleading, as
these errors are caused by client, not server. and the error message
is missing in the HTTP error message when performing the translation.

in this change, we handle the `runtime_error`, and translate them
into `httpd::bad_param_exception`, so that the client can have
HTTP 400 (Bad Request) instead of HTTP 500 (Internal Server Error),
and with informative error message.

for instance, if we apply repair with "small_table_optimization" enabled
on a keyspace with tablets enabled. we should have an HTTP error 400
with "The small_table_optimization option is not supported for tablet repair"
as the body of the error. this would much more helpful.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-04-26 14:25:15 +08:00
Kefu Chai
9de9f401a1 repair: change runtime_error to invalid_argument in do_repair_start()
if an error is caused by the option provided by user, would be better
to throw an `std::invalid_argument` instead of `std::runtime_error`,
so that the caller can make a better decision when handling the
thrown exceptions.

so, in this change, we change the exceptions raise directly in
`repair_service::do_repair_start()` from `std::runtime_error` to
`std::invalid_argument`. please note, in the lambda named `host2ip`,
since the hostname is not provided by user, so we are not changing
the exception type in that lambda.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-04-26 14:24:45 +08:00
Kefu Chai
d737ba1ab2 api/storage_service: coroutinize set_repair()
before this change, `set_repair()` uses a lambda for handling
the client-side requests. and this works great. but the underlying
`repair_start()` throws if any of the given options is not sane.
and we don't handle any of these throw exceptions in `set_repair()`,
from client's point of view, it would get an HTTP 500 error code,
which implies an "Internal Server Error". but actually, we should
blame the client for the error, not the server.

so, to prepare the error handling, let's take the opportunity to
coroutinize the lambda handling the request, so that we can handle
the exception in a more elegant way.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-04-26 14:24:03 +08:00
Michał Jadwiszczak
7f839f727e docs/cql: update ALTER TABLE docs 2024-04-26 07:01:08 +02:00
Michał Jadwiszczak
7cbce78480 test/cqlpytest: add test for prepared ALTER TABLE ... DROP ... USING TIMESTAMP ? 2024-04-26 07:01:02 +02:00
Botond Dénes
7cbe5c78b4 install.sh: use the native nodetool directly
* tools/java b810e8b00e...4ee15fd9ea (1):
  > install.sh: don't install nodetool into /usr/bin

Add a bin/nodetool and install it to bin/ in install.sh. This script
simply forwards to scylla nodetool and it is the replacement for the
Java nodetool, which is dropped from the java-tools's install.sh, in the
submodule update also included in this patch.
With this change, we now hardwire the usage of the native nodetool, as
*the* nodetool, with the intermediary nodetool wrapper script removed
from the picture.
Bash completion was copied from the java tools repository and it is now
installed by the scylla package, together with nodetool.

The Java nodetool is still available as as a fall-back, in case the
native nodetool has problems, at the path of
/opt/scylladb/share/cassandra/bin/nodetool.

Testing

I tested upgrades on a DEB and RPM distro: Ubuntu and Fedora.
First I installed scylla-5.4, then I installed the packages for this PR.
On Ubuntu, I had to use dpkg -i --auto-deconfigure, otherwise, dpkg would
refuse to install the new packages because they break the old ones. No
extra flags were required on Fedora.
In both cases, /usr/bin/nodetool was changed from a thunk calling the
Java nodetool (from 5.4) to the native launcher script from this PR.
/opt/scylladb/share/cassandra/bin/nodetool remained in place and still
works after the upgrade.

I also verified that --nonroot installs also work. Nodetool works both
when called with an absolute path, or when ~/scylladb/bin is added to
$PATH.

Fixes: #18226
Fixes: #17412

Closes scylladb/scylladb#18255

[avi: reset submodule to actual hash we ended up with]
2024-04-25 22:52:00 +03:00
Michał Jadwiszczak
27a4331dcd test/cql-pytest: remove xfail from alter table with timestamp tests
Previous patch introduced `ALTER TABLE ... DROP .. USING TIMESTAM ...`
so those test should no longer fail.

Refs #9929
2024-04-25 21:27:40 +02:00
Michał Jadwiszczak
80f0357436 cql3/statements: extend ALTER TABLE ... DROP to allow specifying timestamp of column drop 2024-04-25 21:27:40 +02:00
Michał Jadwiszczak
7dc0d068c0 cql3/statements: pass query_options to prepare_schema_mutations()
The object is needed to get timestamp from attributes (in a case when
the statement was prepared with parameter marker).
2024-04-25 21:27:40 +02:00
Michał Jadwiszczak
998a65a4f6 cql3/statements: add bound terms to alter table statement
Until now, alter table couldn't take any parameter marker, so the bound
terms were always 0.
Adding `USING TIMESTAMP` to `ALTER TABLE ... DROP` also adds possibility
to prepare a alter table statement with a paramenter marker.
2024-04-25 21:27:40 +02:00
Michał Jadwiszczak
d268641c27 cql3/statements: split alter_table_statement into raw and prepared
Currently alter table doesn't prepare any parameters so raw statement
and prepared one could be the same class.
Later commit will add attributes to the statement, which needs to be
prepared, that's why I'm splitting.
2024-04-25 21:27:40 +02:00
Michał Jadwiszczak
1c5563ba44 schema: allow to specify timestamp of dropped column
In order to drop a column with specified timestamp, we need to
allow it in out schema class.
2024-04-25 21:27:40 +02:00
Avi Kivity
c2b8ca7d71 Merge 'cql3: statements: change default tombstone_gc mode for tablets' from Aleksandra Martyniuk
Repair may miss some tablets that migrated across nodes.
So if tombstones expire after some timeout, then we can
have data resurrection.

Set default tombstone_gc mode to "repair" for tables which
use tablets (if repair is required).

Fixes: #16627.

Closes scylladb/scylladb#18013

* github.com:scylladb/scylladb:
  test: check default value of tombstone_gc
  test: topology: move some functions to util.py
  cql3: statements: change default tombstone_gc mode for tablets
2024-04-25 19:18:37 +03:00