Commit Graph

11801 Commits

Author SHA1 Message Date
Piotr Smaron
47acdc1f98 cql: validate RF change for new DCs in ALTER tablets KS
ALTER tablets KS validated if RF is not changed by more than 1 for DCs
that already had replicas, but not for DCs that didn't have them yet, so
specifying an RF jump from 0 to 2 was possible when listing a new DC in
ALTER tablets KS statement, which violated internal invariants of
tablets load balancer.
This PR fixes that bug and adds a multi-dc testcases to check if adding
replicas to a new DC and removing replicas from a DC is honoring the RF
change constraints.

Refs: #20039
2024-10-07 16:02:01 +02:00
Piotr Smaron
9c5950533f cql: extend test_alter_tablet_keyspace_rf
Added cases to also test decreasing RF and setting the same RF.
Also added extra explanatory comments.
2024-10-07 16:02:00 +02:00
Piotr Smaron
adf453af3f cql: refactor test_tablets::test_alter_tablet_keyspace
1. Renamed the testcase to emphasize that it only focuses on testing
   changing RF - there are other tests that test ALTER tablets KS
in general.
2. Fixed whitespaces according to PEP8
2024-10-07 16:02:00 +02:00
Piotr Smaron
042825247f cql: remove unused helper function from test_tablets
`change_default_rf` is not used anywhere, moreover it uses
`replication_factor` tag, which is forbidden in ALTER tablets KS
statement.
2024-10-07 16:02:00 +02:00
Nikos Dragazis
7a1ec3aa41 test: Test scrub/validate with SSTables from Cassandra
All current unit tests for scrub in validate mode generate random
SSTables on the fly.

Add some more tests with frozen Cassandra SSTables from the source tree
to verify compatibility with Cassandra. Use some of the existing 3.x
Cassandra SSTables to test the valid case, and use the same schema to
generate some corrupted SSTables for the invalid case. Overall, the new
tests cover the following scenarios:

* valid compressed/uncompressed
* compressed/uncompressed with invalid checksums
* compressed/uncompressed with invalid digest

For the compressed SSTable with invalid checksums, a small chunk length
was used (4KiB) to have more chunks with less disk space. For
uncompressed SSTables the chunk length is not configurable.

Finally, since the SSTables live in the source tree, the quarantine
mechanism was disabled.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2024-10-07 15:21:38 +03:00
Nikos Dragazis
5f2be2924e test: Make random schema optional in scrub_test_framework
The scrub_test_framework, which is the foundation for all scrub-related
tests, always generates a random schema upon initialization and makes it
available to the user. This is useful for running tests with ephemeral
SSTables, but is redundant when the creation of the SSTable predates the
test (e.g., it lives in the source tree).

Turn scrub_test_framework into a template with a boolean parameter to
optionally switch off the random schema generation. Also, add an
overload for run() to support passing a ready-to-use SSTable instead of
mutation fragments.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2024-10-07 15:21:38 +03:00
Nikos Dragazis
07ed0a48aa test: Add tests for invalid digests
In a previous patch we extended the validation path of the SSTable layer
to validate the digests along with the checksums.

Add two tests for compressed and uncompressed SSTables to test the
validation API against SSTables with valid checksums but corrupted
digests.

Add two more tests to ensure that the absence of digest does not affect
checksum validation.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2024-10-07 15:21:38 +03:00
Nikos Dragazis
39a74fb692 test: Merge scrub/validate tests for compressed and uncompressed cases
Currently, every scrub/validate test is duplicated to cover both
compressed and uncompressed SSTables. However, except for the
compression type, the tests are identical. This leads to some code
bloat.

Introduce common functions parameterized by the compression type to
reduce code duplication. Also, group together the compressed and
uncompressed variants into one compression-agnostic test.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2024-10-07 15:21:38 +03:00
Avi Kivity
7dad248ac7 Merge 'Fix sstables registry mock' from Pavel Emelyanov
There are two issues in it. First, listing the registry with a consumer callback passes wrong argument to the consumer. Second, the primary key of the registry is wrong. Both issues don't show up, because existing tests that use mock don't read from it, only write. Tests that read from registry are python tests that start scylla and thus use real registry.

Closes scylladb/scylladb#20946

* github.com:scylladb/scylladb:
  test: Use corrcet key in sstables registry mock
  test: Pass entry status to mock registry consumer
2024-10-07 13:56:26 +03:00
Nadav Har'El
987042be68 mv, test: reproduce missing validation for view name
This patch adds reproducer tests (still failing) for issue #20755, which
is about missing validation of materialized view names:

1. Unlike table and keyspace names which are limited to 48 characters,
   we forgot to limit view name length, and an excessively long name can
   cause Scylla to shut down :-(

2. Unlike table and keyspace names which only allow alphanumeric
   characters, view names are missing this check and can include any
   characters.

3. Luckily, even though we are missing the alphanumeric check, we at
   least don't allow "/" in view names (if we allowed them, it could
   allow users to write in any directory in the filesystem!). But when
   this happens, we get an internal error instead of the expected
   errors.

The first test also fails on Cassandra (it doesn't crash it, but leaves
the table in a strange state), but the other two pass.

Refs #20755

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes scylladb/scylladb#20761
2024-10-07 13:49:58 +03:00
Pavel Emelyanov
1870873538 test: Fix test_multiple_data_dirs
The one was broken from the very beginning. It only checked that after
creating a table, its directory is created in all datadirs. But it
didn't check that after restart populating happens from the all. That's
because all directories by 0th were always empty, so not-populating from
them didn't skip any data.

Fix it by moving all sstables from datadirs[0] to datadirs[1] before
restart. With that update not-populating data from datadirs[1] will
be noticed instantly. Fortunately, previous patches fixed that, so the
test still passes.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2024-10-07 12:04:23 +03:00
Benny Halevy
fa8fe62e90 test: cql-pytest: test_compaction: add test_compactionstats_after_major_compaction
Test that compactionstats are empty, i.e.
there are no required compactions following
major compaction.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-10-07 10:24:06 +03:00
Benny Halevy
630b792bd0 test/cql-pytest: rename test_compaction{_tombstone_gc,}
Prepare to add more tests related to compaction to
this test suite.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-10-07 10:18:30 +03:00
Botond Dénes
af124993a4 Merge 'Do not remove objects from backup storage after restore' from Pavel Emelyanov
The restore-from-s3 task uses load-and-stream internally which, in turn, unlinks loaded sstables on success. That's not what user expects when it restores from backup, objects should remain in bucket afterwards.

Closes scylladb/scylladb#20947

* github.com:scylladb/scylladb:
  test: Add check that restored-from objects are not removed
  sstables_loader: Dont unlink sstables when restoring from S3
  sstables_loader: Make primary_replica_only bool_class RAII field
2024-10-04 14:59:40 +03:00
Pavel Emelyanov
6b480589fe Merge 'treewide: accept list of sstables in "restore" API ' from Kefu Chai
before this change, we enumerate the sstables tracked by the
system.sstables table, and restore them when serving
requests to "storage_service/restore" API. this works fine with
"storage_service/backup" API. but this "restore" API cannot be
used as a drop-in replacement of the rclone based API currently
used by scylla-manager.

in order to fill the gap, in this change:

* add the "prefix" parameter for specifying the shared prefix of
  sstables
* add the "sstables" parameter for specifying the list of  TOC
  components of sstables
* remove the "snapshot" parameter, as we don't encode the prefix
  on scylla's end anymore.
* make the "table" parameter mandatory.

Fixes https://github.com/scylladb/scylladb/issues/20461

----

this change is a part of the efforts to bring the native backup/restore to scylla, no need to backprt.

Closes scylladb/scylladb#20685

* github.com:scylladb/scylladb:
  treewide: accept list of sstables in "restore" API
  sstable: pass get_storage_option to sstable_directory::load_sstable()
  test/nodetool: add body parameter to `expected_request`
  tools/scylla-nodetool: enable nodetool to write HTTP body
2024-10-04 12:38:08 +03:00
Botond Dénes
07094c3e44 Merge 'replica: Fix tombstone GC during tablet split preparation' from Raphael "Raph" Carvalho
During split prepare phase, there will be more than 1 compaction group with
overlapping token range for a given replica.

Assume tablet 1 has sstable A containing deleted data, and sstable B containing
a tombstone that shadows data in A.

Then split starts:
1) sstable B is split first, and moved from main (unsplit) group to a
split-ready group
2) now compaction runs in split-ready group before sstable A is split

tombstone GC logic today only looks at underlying group, so compaction is step
2 will discard the deleted data in A, since it belongs to another group (the
unsplit one), and so the tombstone can be purged incorrectly.

To fix it, compaction will now work with all uncompacting sstables that belong
to the same replica, since tombstone GC requires all sstables that possibly
contain shadowed data to be available for correct decision to be made.

Fixes https://github.com/scylladb/scylladb/issues/20044.

Branches 6.0, 6.1 and 6.2 are vulnerable, so backport is needed.

Closes scylladb/scylladb#20939

* github.com:scylladb/scylladb:
  replica: Fix tombstone GC during tablet split preparation
  service: Improve error handling for split
2024-10-04 10:29:42 +03:00
Nikos Dragazis
c893f06409 sstables: Add digest check in compressed data source
Following the addition of digest check in the checksummed data source,
add the same feature to the compressed data source as well. This ensures
consistent behavior across any type of SSTable.

This is added as an optional feature so that we can preserve the current
behavior, that is verify only the per-chunk checksums during normal user
reads. To ensure zero cost at runtime when disabled, we introduce the
on/off switch as a template parameter.

The digest calculation for compressed SSTables depends on the SSTable
format, hence the new template argument for the checksum mode. This is
consistent with the compressed data sink.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2024-10-03 18:09:01 +03:00
Tomasz Grabiec
62f3d9e173 perf: perf_fast_forward: Add test case for querying missing rows 2024-10-03 16:26:41 +02:00
Tomasz Grabiec
4602ba90df perf-fast-forward: Allow overriding promoted index block size
For testing dense clustering index.
2024-10-03 16:26:41 +02:00
Tomasz Grabiec
1782456a52 perf-fast-forward: Test subsequent key reads from the middle in test_large_partition_select_few_rows 2024-10-03 16:26:41 +02:00
Tomasz Grabiec
751fa10de8 perf-fast-forward: Allow adding key offset in test_large_partition_select_few_rows 2024-10-03 16:26:41 +02:00
Tomasz Grabiec
10c6990e41 perf-fast-forward: Use single-partition reads in test_large_partition_select_few_rows
It's a more realistic scenario than a full scan.
2024-10-03 16:26:28 +02:00
Kefu Chai
7b82f3a375 test/lib: remove redundant fmt::to_string() in seastar::format()
previously change, implementation was unnecessarily verbose and less
efficient, as it created and immediately discarded temporary strings.
remove unnecessary use of `fmt::to_string()` when arguments are already
being formatted by `seastar::format()`.

in this this change:

- eliminates creation of temporary `std::string` instances
- reduces memory allocations and copies
- improves performance
- simplifies the code

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20923
2024-10-03 15:36:55 +03:00
Tomasz Grabiec
1b82d5117a sstables: bsearch_clustered_cursor: Narrow down range using "end" position of the block
This is optimization.

Example:

  block0: start=aaa, end=aaA
  block1: start=bbb, end=bbB
  block2: whatever

Before the patch, advance_to("aAA") would skip to block0, and upper
bound probe would skip to block1. This way, the reader would read the
range of block0 from the data file.

After the patch, "end" position is taken into account, so
advance_to("aAA") will notice that block0 doesn't contain the position
and will skip to block1. This is especially important for dense
indexes, as it allows us to skip accessing data file if the search key
is missing.

It also solves the edge case problem related to the fact that single
row reads are using a range which with positions which are not equal
to the key, but are before(key) and after(key) for the lower bound and
upper bound respectively. Before the patch, advance_to(before("bbb"))
would skip to block0, before the position is before the block1's
start. And upper bound probe for after("bbb") would point to
block2. This way the read would scan block0 needlessly. After the
patch, advance_to(before("bbb")) will skip to block1 because we notice
based on "end" that block0 doesn't contain the position.

This change also ensures that the start position of the upper bound
entry of the after_key(pos), where pos is the last advance_to()
position, is warm in cache. This is needed to optimize single-row
reads with a dense index so that they always read exactly one promoted
index block. For this to work, probe_upper_bound() for the
after_key(row) always needs to find the upper bound block in
cache.
2024-10-03 14:16:05 +02:00
Tomasz Grabiec
c905554121 test: sstables: sstable_3_x_test: Improve failure message 2024-10-03 14:16:05 +02:00
Tomasz Grabiec
7f077893ed sstables: mx: writer: Never include partition_end marker in promoted index block width
Currently, it may happen that the last promoted index block includes
the partition_end marker. That's because we first write the partition
end marker and then emit the unclosed block. This behavior matches
Cassandra (checked in 3.x and 5.0.1).

This is problematic for ruling out data file reads based on index.
The width field is currently unused, but it will be used later where
the width of the last block is used to compute the skip position past
the last block for lookups which land after all keys in the
partition. If width includes the marker then such a skip would land in
the next partition, which is incorrect, as the reader context expects
a cell element. Even if that was recognized, it's wrong - if this is
not a single partition read (so upper bound is not at the next
partition too), then we would read from the wrong (next) partition.

We want to be able to make such skips in order to avoid unnecessary
data file IO for reads of missing rows. Currently, we would always
read the last block even if the key is past its "end" position.

Another way to solve this would be to propagate the "past the last
block" condition from the index cursor to the reader and let it deal
with it, but the logic for that would be complicated. With this fix,
there is no special logic required.
2024-10-03 14:09:57 +02:00
Pavel Emelyanov
4465bd9e5e test: Add check that restored-from objects are not removed
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2024-10-03 13:37:04 +03:00
Kamil Braun
e67016540c Merge 'Node replace and remove operations: Add deprecate IP addresses usage warning.' from Sergey Zolotukhin
- As part of deprecation of IP address usage, warning messages were added when IP addresses specified in the `ignore-dead-nodes` and `--ignore-dead-nodes-for-replace` options for scylla and nodetool.
- Slight optimizations for `utils::split_comma_separated_list`, ` host_id_or_endpoint lists` and `storage_service` remove node operations, replacing `std::list` usage with `std::vector`.

Fixes scylladb/scylladb#19218

Backport: 6.2 as it's not yet released.

Closes scylladb/scylladb#20756

* github.com:scylladb/scylladb:
  config: Add a warning about use of IP address for join topology and replace operations.
  nodetool: Add IP address usage warning for 'ignore-dead-nodes'.
  tests: Fix incorrect UUIDs in test_nodeops
  utils: Optimizations for utils::split_comma_separated_list and usage of host_id_or_endpoint lists
2024-10-03 11:08:28 +02:00
Kamil Braun
d2233d4400 Merge 'test: update cql/ tests to work with tablets enabled by default' from Konstantin Osipov
Explicitly disable tablets for features which still dont' work with tablets: cdc, lwt, coutners.

Closes scylladb/scylladb#20858

* github.com:scylladb/scylladb:
  test: make cdc tests pass with tablets on by default
  test: make cql/counters* pass with and without tablets
  test: make cql/lwt_* pass with and without tablets
  test: rename cql/list_test to cql/lwt_list_test
2024-10-03 10:53:17 +02:00
Kefu Chai
f9091066b7 treewide: replace boost::irange with std::views::iota where possible
when building scylla with the standard library from GCC-14.2, shipped by
fedora 41, we have following build failure:

```
/home/kefu/.local/bin/clang++ -DDEBUG -DDEBUG_LSA_SANITIZER -DFMT_SHARED -DSANITIZE -DSCYLLA_BUILD_MODE=debug -DSCYLLA_ENABLE_ERROR_INJECTION -DSEASTAR_API_LEVEL=7 -DSEASTAR_DEBUG -DSEASTAR_DEBUG_PROMISE -DSEASTAR_DEBUG_SHARED_PTR -DSEASTAR_DEFAULT_ALLOCATOR -DSEASTAR_LOGGER_COMPILE_TIME_FMT -DSEASTAR_LOGGER_TYPE_STDOUT -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -DSEASTAR_SHUFFLE_TASK_QUEUE -DSEASTAR_SSTRING -DSEASTAR_TYPE_ERASE_MORE -DXXH_PRIVATE_API -DCMAKE_INTDIR=\"Debug\" -I/home/kefu/dev/scylladb -I/home/kefu/dev/scylladb/build/gen -I/home/kefu/dev/scylladb/seastar/include -I/home/kefu/dev/scylladb/build/seastar/gen/include -I/home/kefu/dev/scylladb/build/seastar/gen/src -isystem /home/kefu/dev/scylladb/abseil -g -Og -g -gz -std=gnu++23 -fvisibility=hidden -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wimplicit-fallthrough -Wno-c++11-narrowing -Wno-deprecated-copy -Wno-mismatched-tags -Wno-missing-field-initializers -Wno-overloaded-virtual -Wno-unsupported-friend -Wno-unused-parameter -ffile-prefix-map=/home/kefu/dev/scylladb/build=. -march=x86-64-v3 -mpclmul -Xclang -fexperimental-assignment-tracking=disabled -Werror=unused-result -fstack-clash-protection -fsanitize=address -fsanitize=undefined -MD -MT CMakeFiles/scylla-main.dir/Debug/init.cc.o -MF CMakeFiles/scylla-main.dir/Debug/init.cc.o.d -o CMakeFiles/scylla-main.dir/Debug/init.cc.o -c /home/kefu/dev/scylladb/init.cc
In file included from /home/kefu/dev/scylladb/init.cc:12:
In file included from /home/kefu/dev/scylladb/db/config.hh:20:
In file included from /home/kefu/dev/scylladb/locator/abstract_replication_strategy.hh:26:
/home/kefu/dev/scylladb/locator/tablets.hh:410:30: error: unexpected type name 'size_t': expected expression
  410 |         return boost::irange<size_t>(0, tablet_count()) | boost::adaptors::transformed([] (size_t i) {
      |                              ^
/home/kefu/dev/scylladb/locator/tablets.hh:410:23: error: no member named 'irange' in namespace 'boost'
  410 |         return boost::irange<size_t>(0, tablet_count()) | boost::adaptors::transformed([] (size_t i) {
      |                ~~~~~~~^
/home/kefu/dev/scylladb/locator/tablets.hh:410:38: error: left operand of comma operator has no effect [-Werror,-Wunused-value]
  410 |         return boost::irange<size_t>(0, tablet_count()) | boost::adaptors::transformed([] (size_t i) {
      |                                      ^
3 errors generated.
[16/782] Building CXX object CMakeFiles/scylla-main.dir/Debug/keys.cc.o
[17/782] Building CXX object CMakeFiles/scylla-main.dir/Debug/counters.cc.o
[18/782] Building CXX object CMakeFiles/scylla-main.dir/Debug/partition_slice_builder.cc.o
[19/782] Building CXX object CMakeFiles/scylla-main.dir/Debug/mutation_query.cc.o
FAILED: CMakeFiles/scylla-main.dir/Debug/mutation_query.cc.o
/home/kefu/.local/bin/clang++ -DDEBUG -DDEBUG_LSA_SANITIZER -DFMT_SHARED -DSANITIZE -DSCYLLA_BUILD_MODE=debug -DSCYLLA_ENABLE_ERROR_INJECTION -DSEASTAR_API_LEVEL=7 -DSEASTAR_DEBUG -DSEASTAR_DEBUG_PROMISE -DSEASTAR_DEBUG_SHARED_PTR -DSEASTAR_DEFAULT_ALLOCATOR -DSEASTAR_LOGGER_COMPILE_TIME_FMT -DSEASTAR_LOGGER_TYPE_STDOUT -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -DSEASTAR_SHUFFLE_TASK_QUEUE -DSEASTAR_SSTRING -DSEASTAR_TYPE_ERASE_MORE -DXXH_PRIVATE_API -DCMAKE_INTDIR=\"Debug\" -I/home/kefu/dev/scylladb -I/home/kefu/dev/scylladb/build/gen -I/home/kefu/dev/scylladb/seastar/include -I/home/kefu/dev/scylladb/build/seastar/gen/include -I/home/kefu/dev/scylladb/build/seastar/gen/src -isystem /home/kefu/dev/scylladb/abseil -g -Og -g -gz -std=gnu++23 -fvisibility=hidden -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wimplicit-fallthrough -Wno-c++11-narrowing -Wno-deprecated-copy -Wno-mismatched-tags -Wno-missing-field-initializers -Wno-overloaded-virtual -Wno-unsupported-friend -Wno-unused-parameter -ffile-prefix-map=/home/kefu/dev/scylladb/build=. -march=x86-64-v3 -mpclmul -Xclang -fexperimental-assignment-tracking=disabled -Werror=unused-result -fstack-clash-protection -fsanitize=address -fsanitize=undefined -MD -MT CMakeFiles/scylla-main.dir/Debug/mutation_query.cc.o -MF CMakeFiles/scylla-main.dir/Debug/mutation_query.cc.o.d -o CMakeFiles/scylla-main.dir/Debug/mutation_query.cc.o -c /home/kefu/dev/scylladb/mutation_query.cc
In file included from /home/kefu/dev/scylladb/mutation_query.cc:12:
In file included from /home/kefu/dev/scylladb/schema/schema_registry.hh:17:
In file included from /home/kefu/dev/scylladb/replica/database.hh:11:
In file included from /home/kefu/dev/scylladb/locator/abstract_replication_strategy.hh:26:
/home/kefu/dev/scylladb/locator/tablets.hh:410:30: error: unexpected type name 'size_t': expected expression
  410 |         return boost::irange<size_t>(0, tablet_count()) | boost::adaptors::transformed([] (size_t i) {
      |                              ^
/home/kefu/dev/scylladb/locator/tablets.hh:410:23: error: no member named 'irange' in namespace 'boost'
  410 |         return boost::irange<size_t>(0, tablet_count()) | boost::adaptors::transformed([] (size_t i) {
      |                ~~~~~~~^
/home/kefu/dev/scylladb/locator/tablets.hh:410:38: error: left operand of comma operator has no effect [-Werror,-Wunused-value]
  410 |         return boost::irange<size_t>(0, tablet_count()) | boost::adaptors::transformed([] (size_t i) {
      |                                      ^
In file included from /home/kefu/dev/scylladb/mutation_query.cc:12:
In file included from /home/kefu/dev/scylladb/schema/schema_registry.hh:17:
In file included from /home/kefu/dev/scylladb/replica/database.hh:37:
In file included from /home/kefu/dev/scylladb/db/snapshot-ctl.hh:20:
/home/kefu/dev/scylladb/tasks/task_manager.hh:403:54: error: no member named 'irange' in namespace 'boost'
  403 |         co_await coroutine::parallel_for_each(boost::irange(0u, smp::count), [&tm, id, &res, &func] (unsigned shard) -> future<> {
      |                                               ~~~~~~~^
4 errors generated.
```

so let's take the opportunity to switch from `boost::irange` to
`std::views::iota`.

in this change, we:

- switch from boost::irange to std::views::iota for better standard library compatibility
- retain boost::irange where step parameter is used, as std::views::iota doesn't support it
- this change partially modernizes our range usage while maintaining
- existing functionality

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20924
2024-10-03 10:33:33 +03:00
Pavel Emelyanov
1da1d131b2 test: Use corrcet key in sstables registry mock
The "real" registry defines its primary key as (location, generation)
pair, where location is the partition key and generation is clustering
key. The registry mock uses only location part as primary key, while it
must use both.

The buggy mock works simply because the listing API is in fact not used
by unit tests. Those tests that do need it are python tests that start
scylla and thus implicitly use real registry.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2024-10-03 10:07:11 +03:00
Pavel Emelyanov
a503e2ab10 test: Pass entry status to mock registry consumer
When sstables registry is listed, the passed consumer accepts entry
status as its first argument, not its location (location is passed as a
search key)

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2024-10-03 10:06:28 +03:00
Piotr Dulikowski
6778001313 Merge 'cql3: Make creating MV respect ID option' from Dawid Mędrek
Before these changes, we could create a materialized
view specifying its ID, but the option was ignored.

This commit makes Scylla respect the option. Now specifying
the ID results in the MV being created with that specific ID.
This way, Scylla's behavior is consistent with Cassandra's.

Because Cassandra doesn't mention the option in its
user documentation, we don't update it either in case
the semantics of it changes in the future -- we want
to have an open door for any modifications.

Note that Cassandra returns a server error if the provided
ID is already in use, both in the case of regular tables
and MVs. That's most likely a bug. Instead of following that
behavior, we stay consistent with the current semantics of
creating a regular table in Scylla: if the provided ID is
already used, return an InvalidRequest.

The last thing worth pointing out is Cassandra handles
`WITH ID = null` as a special case; normally, specifying
an invalid ID results in a ConfigurationException, but a null
is treated as a syntax error. As in the previous paragraph,
we stay consistent with the semantics of regular tables and
all invalid IDs, null included, lead to a ConfigurationException.

We also add a few short tests verifying that the implementation
works as intended.

Fixes scylladb/scylladb#20616

Backport not needed: the semantics of the option was never documented in either Cassandra, or Scylla.

Closes scylladb/scylladb#20773

* github.com:scylladb/scylladb:
  test/cql-pytest: Get rid of unnecessary processing describe statements
  cql3: Make creating MV respect ID option
2024-10-03 08:31:07 +02:00
Ferenc Szili
cdf775d3cc test: test tombstone GC disabled on pending replica
This tests if tombstone GC is disabled on pending replicas
2024-10-02 16:37:57 +02:00
Raphael S. Carvalho
93815e0649 replica: Fix tombstone GC during tablet split preparation
During split prepare phase, there will be more than 1 compaction group with
overlapping token range for a given replica.

Assume tablet 1 has sstable A containing deleted data, and sstable B containing
a tombstone that shadows data in A.

Then split starts:
1) sstable B is split first, and moved from main (unsplit) group to a
split-ready group
2) now compaction runs in split-ready group before sstable A is split

tombstone GC logic today only looks at underlying group, so compaction is step
2 will discard the deleted data in A, since it belongs to another group (the
unsplit one), and so the tombstone can be purged incorrectly.

To fix it, compaction will now work with all uncompacting sstables that belong
to the same replica, since tombstone GC requires all sstables that possibly
contain shadowed data to be available for correct decision to be made.

Fixes #20044.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-10-02 11:26:13 -03:00
Konstantin Osipov
1d1777b13a test: make cdc tests pass with tablets on by default
CDC is not supported with tablets, explicitly disable tablets
in CDC keyspace definition.
2024-10-02 06:37:14 -04:00
Konstantin Osipov
0e3dbec277 test: make cql/counters* pass with and without tablets
Counters are not supported with tablets, make sure
the test works in any ScyllaDB configuration.
2024-10-02 06:37:14 -04:00
Konstantin Osipov
92aca17bc5 test: make cql/lwt_* pass with and without tablets
Lightweight transactions don't support tablets, so let's
explicitly disable tablets in LWT tests.
2024-10-02 06:37:14 -04:00
Konstantin Osipov
7c64fc0c4f test: rename cql/list_test to cql/lwt_list_test
This test is actually testing lists with LWT, so should
have the corresponding name. Going forward we'll patch CQL LWT
tests for tablets, so let's group them together.
2024-10-02 06:37:14 -04:00
Sergey Zolotukhin
a871321ecf tests: Fix incorrect UUIDs in test_nodeops
It was found that the UUIDs used in test_nodeops were
invalid. This update replaces those UUIDs with newly generated
random UUIDs.
2024-10-02 11:56:59 +02:00
Dawid Mędrek
de88c150f6 test/cql-pytest: Get rid of unnecessary processing describe statements
As part of scylladb/scylladb@d42f160, we added a test verifying that
restoring the schema works as intended. Unfortunately, because of
scylladb/scylladb#20616, we had to manually process the results
of `DESCRIBE SCHEMA` to exclude the ID parameter and be able to
compare restore statements corresponding to the same view.

Now that materialized views respect the ID parameter, we can get rid
of that logic.
2024-10-01 22:04:05 +02:00
Dawid Mędrek
552c752005 cql3: Make creating MV respect ID option
Before these changes, we could create a materialized
view specifying its ID, but the option was ignored.

This commit makes Scylla respect the option. Now specifying
the ID results in the MV being created with that specific ID.
This way, Scylla's behavior is consistent with Cassandra's.

Because Cassandra doesn't mention the option in its
user documentation, we don't update it either in case
the semantics of it changes in the future -- we want
to have an open door for any modifications.

Note that Cassandra returns a server error if the provided
ID is already in use, both in the case of regular tables
and MVs. That's most likely a bug. Instead of following that
behavior, we stay consistent with the current semantics of
creating a regular table in Scylla: if the provided ID is
already used, return an InvalidRequest.

The last thing worth pointing out is Cassandra handles
`WITH ID = null` as a special case; normally, specifying
an invalid ID results in a ConfigurationException, but a null
is treated as a syntax error. As in the previous paragraph,
we stay consistent with the semantics of regular tables and
all invalid IDs, null included, lead to a ConfigurationException.

We also add a few short tests verifying that the implementation
works as intended.
2024-10-01 22:03:58 +02:00
Kefu Chai
9b5eab0dde test/lib: include <fmt/std.h> for formatting std::optional
before this change, when compiling with fmtlib v11.0.2 and clang
v19.1.0, the compiler fails like:

```
/usr/bin/clang++ -DBOOST_REGEX_DYN_LINK -DBOOST_REGEX_NO_LIB -DBOOST_UNIT_TEST_FRAMEWORK_DYN_LINK -DBOOST_UNIT_TEST_FRAMEWORK_NO_LIB -DDEBUG -DDEBUG_LSA_SANITIZER -DFMT_SHARED -DSANITIZE -DSCYLLA_BUILD_MODE=debug -DSCYLLA_ENABLE_ERROR_INJECTION -DSEASTAR_API_LEVEL=7 -DSEASTAR_DEBUG -DSEASTAR_DEBUG_PROMISE -DSEASTAR_DEBUG_SHARED_PTR -DSEASTAR_DEFAULT_ALLOCATOR -DSEASTAR_LOGGER_COMPILE_TIME_FMT -DSEASTAR_LOGGER_TYPE_STDOUT -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -DSEASTAR_SHUFFLE_TASK_QUEUE -DSEASTAR_SSTRING -DSEASTAR_TYPE_ERASE_MORE -DXXH_PRIVATE_API -DCMAKE_INTDIR=\"Debug\" -I/home/kefu/dev/scylladb -I/home/kefu/dev/scylladb/build/gen -I/home/kefu/dev/scylladb/seastar/include -I/home/kefu/dev/scylladb/build/seastar/gen/include -I/home/kefu/dev/scylladb/build/seastar/gen/src -I/home/kefu/dev/scylladb/build -isystem /home/kefu/dev/scylladb/abseil -isystem /home/kefu/dev/scylladb/build/rust -g -Og -g -gz -std=gnu++23 -fvisibility=hidden -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wimplicit-fallthrough -Wno-c++11-narrowing -Wno-deprecated-copy -Wno-mismatched-tags -Wno-missing-field-initializers -Wno-overloaded-virtual -Wno-unsupported-friend -Wno-enum-constexpr-conversion -Wno-unused-parameter -ffile-prefix-map=/home/kefu/dev/scylladb/build=. -march=x86-64-v3 -mpclmul -Xclang -fexperimental-assignment-tracking=disabled -Werror=unused-result -fstack-clash-protection -fsanitize=address -fsanitize=undefined -MD -MT test/lib/CMakeFiles/test-lib.dir/Debug/cql_assertions.cc.o -MF test/lib/CMakeFiles/test-lib.dir/Debug/cql_assertions.cc.o.d -o test/lib/CMakeFiles/test-lib.dir/Debug/cql_assertions.cc.o -c /home/kefu/dev/scylladb/test/lib/cql_assertions.cc
In file included from /home/kefu/dev/scylladb/test/lib/cql_assertions.cc:12:
In file included from /usr/include/fmt/ranges.h:20:
In file included from /usr/include/fmt/format.h:41:
/usr/include/fmt/base.h:2673:45: error: implicit instantiation of undefined template 'fmt::detail::type_is_unformattable_for<std::vector<std::optional<seastar::basic_sstring<signed char, unsigned int, 31, false>>>, char>'
 2673 |     type_is_unformattable_for<T, char_type> _;
      |                                             ^
/usr/include/fmt/base.h:2735:23: note: in instantiation of function template specialization 'fmt::detail::parse_format_specs<std::vector<std::optional<seastar::basic_sstring<signed char, unsigned int, 31, false>>>, fmt::detail::compile_parse_context<char>>' requested here
 2735 |         parse_funcs_{&parse_format_specs<Args, parse_context_type>...} {}
      |                       ^
/usr/include/fmt/base.h:2884:47: note: in instantiation of member function 'fmt::detail::format_string_checker<char, int, std::vector<std::optional<seastar::basic_sstring<signed char, unsigned int, 31, false>>>, std::vector<std::optional<managed_bytes>>>::format_string_checker' requested here
 2884 |       detail::parse_format_string<true>(str_, checker(s));
      |                                               ^
/home/kefu/dev/scylladb/test/lib/cql_assertions.cc:132:34: note: in instantiation of function template specialization 'fmt::basic_format_string<char, int &, std::vector<std::optional<seastar::basic_sstring<signed char, unsigned int, 31, false>>> &, const std::vector<std::optional<managed_bytes>> &>::basic_format_string<char[35], 0>' requested here
  132 |             fail(seastar::format("row {} differs, expected {} got {}", row_nr, row, actual));
      |                                  ^
/usr/include/fmt/base.h:1616:8: note: template is declared here
 1616 | struct type_is_unformattable_for;
      |        ^
/home/kefu/dev/scylladb/test/lib/cql_assertions.cc:132:34: error: call to consteval function 'fmt::basic_format_string<char, int &, std::vector<std::optional<seastar::basic_sstring<signed char, unsigned int, 31, false>>> &, const std::vector<std::optional<managed_bytes>> &>::basic_format_string<char[35], 0>' is not a constant expression
  132 |             fail(seastar::format("row {} differs, expected {} got {}", row_nr, row, actual));
      |                                  ^
```

because the formatter for `std::optional<>` is defined in fmt/std.h.

so, in this change, we include the used header.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20922
2024-10-01 22:32:16 +03:00
Kefu Chai
787ea4b1d4 treewide: accept list of sstables in "restore" API
before this change, we enumerate the sstables tracked by the
system.sstables table, and restore them when serving
requests to "storage_service/restore" API. this works fine with
"storage_service/backup" API. but this "restore" API cannot be
used as a drop-in replacement of the rclone based API currently
used by scylla-manager.

in order to fill the gap, in this change:

* add the "prefix" parameter for specifying the shared prefix of
  sstables
* add the "sstables" parameter for specifying the list of  TOC
  components of sstables
* remove the "snapshot" parameter, as we don't encode the prefix
  on scylla's end anymore.
* make the "table" parameter mandatory.

Fixes scylladb/scylladb#20461
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-10-01 23:24:56 +08:00
Kefu Chai
283697e316 test/nodetool: add body parameter to expected_request
before this change, `expected_request` only includes query strings
for the parameters of requests. but we will add an API
("storage_service/restore") which accepts its parameters in HTTP body
as well.

in this change, we add an optional `body` member to `expected_request`,
so that we can mock the APIs which pass the parameters with the HTTP
body.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-10-01 23:24:56 +08:00
Benny Halevy
5a0f3889e0 treewide: use std::ranges sort functions rather than boost
Using the standard library is preffered over boost.

In cql3/expr/expression.cc to_sorted_vector got more of a
face-list and was modernized to use also std::unique
and while at it, to move its input range in the uniquely sorted
result vector.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-10-01 14:19:05 +03:00
Avi Kivity
e99426df60 treewide: de-static namespace scope functions in headers
'static inline' is always wrong in headers - if the same header is
included multiple times, and the function happens not to be inlined,
then multiple copies of it will be generated.

Fix by mechanically changing '^static inline' to 'inline'.
2024-10-01 14:02:50 +03:00
Avi Kivity
e9425e15b2 treewide: remove dependency on boost asio address_v4
It's not used. There's a comment mentioning it prevents some type
conflict, but apparently that was fixed some time ago.

Closes scylladb/scylladb#20883
2024-10-01 14:00:50 +03:00
Pavel Emelyanov
24598848a9 Merge 'virtual_tables: snapshots: include all snapshots' from Benny Halevy
Use database::get_snapshot_details to get the details
of all snapshots on disk, in particular those of
deleted tables.

Add test_snapshots_dropped_table to test listing
of snapshots of a deleted table.
And harden the existing test cases to use a unique
snapshot tag and to delete it when the test ends.

Fixes #18313

* No backport required at this time since this is rather minor UX issue that weren't hit in the field AFAIK

Closes scylladb/scylladb#20869

* github.com:scylladb/scylladb:
  cql-pytest: test_virtual_tables: add test_snapshots_multiple_keyspaces
  virtual_tables: snapshots: include all snapshots
2024-10-01 13:56:13 +03:00
Gleb Natapov' via ScyllaDB development
22368b13f2 api: introduce raft stepdown REST API
Also provide test.py util function to trigger it. Can be useful for
testing.
2024-10-01 12:18:49 +02:00