Commit Graph

45758 Commits

Author SHA1 Message Date
Tomasz Grabiec
bf18a17bd6 tablets: scheduler: Fix temporary imbalance in a mixed-capacity cluster on decommission
When tablet scheduler drains nodes, it chooses target location based
on "badness" metric. Nodes with lowest score are preferred. Before the
patch, the score which was used was the number of tablets on that node
post-movement. This way we populate least-loaded node first. But this
works only if nodes have equal number of shards. If nodes have different
capacity, then number of tablets is not a good metric, because we don't
aim to equalize per-node count, but per-shard count. We assume that each
shard has equal capacity.

Because of this bug, during decommission, the nodes with fewer shards
would be preferred to receive replicas, which may lead to overloading
of those nodes. This imbalance would be later fixed by the normal load
balancing logic, but it's still problematic.

Fixes #21783

Closes scylladb/scylladb#21860
2024-12-10 14:18:03 +02:00
Botond Dénes
5d040e0206 Merge 'truncate: commit log replay positions are not saved correctly' from Ferenc Szili
TRUNCATE TABLE saves the current commit log replay positions in case there is a crash so that replay knows where to begin replaying the mutations. These are collected and saved per shard into `system.truncated`. In case a shard received no mutations, its replay position will be an empty, default constructed object of type `db::replay_position` with its members set to 0. Truncate will incorrectly interpret these empty replay positions as if they were coming from shard 0, and save them as such, potentially overwriting an actual valid replay position coming from the actual shard 0. In the case of a crash, this will cause the commit log on shard 0 to be replayed from the beginning, and result with data resurrection.

Fixes #21719

Closes scylladb/scylladb#21722

* github.com:scylladb/scylladb:
  test: add test for truncate saving replay positions
  database: correctly save replay position for truncate
2024-12-10 10:05:30 +02:00
Botond Dénes
924189c50e Merge 'replica/table: improve error message when encountering orphaned sstables' from Lakshmi Narayanan Sreethar
On startup, if a server reads an sstable that belongs to a tablet that
doesn't have any local replica, it throws an error in the following
format and refuses to start :

```
Storage wasn't found for tablet 1 of table test.test
```

This patch updates the code path to throw a nicer error that includes
the sstable name that caused the problem.

This patch also adds a testcase to verify the error being thrown.

Fixes https://github.com/scylladb/scylladb/issues/18038

PR improves an error message - no need to backport.

Closes scylladb/scylladb#21805

* github.com:scylladb/scylladb:
  replica/table: fix indent in compaction_group_for_sstable
  replica/table: improve error message when encountering orphaned sstables
2024-12-10 06:34:12 +02:00
Kefu Chai
ce2f80c227 treewide: migrate from boost::make_iterator_range to ranges::subrange
Replace boost::make_iterator_range() with std::ranges::subrange.

This change improves code modernization and reduces external dependencies:

- Replace boost::make_iterator_range() with std::ranges::subrange
- Remove boost/range/iterator_range.hpp include
- Improve iterator type detection in interval.hh using std::ranges::const_iterator_t<Range>

This is part of ongoing efforts to modernize our codebase and minimize
external dependencies.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21787
2024-12-09 21:31:53 +02:00
Pavel Emelyanov
6eb6b96456 dirty-memory-manager: Brush up "blocked" state check
One of run_when_memory_available() checks mirrors the one done by the
execution_permitted() helper, so its worth re-using it. Since the former
helper is header template, the latter is worth moving to header too.
And, once re-used, the `bool blocking` variable becomes excessive, and
the `if (blocking)` check can also be expressed with fewer LOCs.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>

Closes scylladb/scylladb#21812
2024-12-09 20:44:22 +02:00
Kefu Chai
48c8d24345 treewide: drop support for fmt < v10
since fedora 38 is EOL. and fedora 39 comes with fmt v10.0.0, also,
we've switched to the build image based on fedora 40, which ships
fmt-devel v10.2.1, there is no need to support fmt < 10.

in this change, we drop the support fmt < 10.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21847
2024-12-09 20:42:38 +02:00
Avi Kivity
1bac6b75dc Merge 'Reserve IOCBs for tool applications' from Botond Dénes
Artifact tests have been failing since the switch to the native nodetool, because ScyllaDB doesn't leave any IOCBs for tools. On some setups it will consume all of them and then nodetool and any other native app will refuse to start because it will fail to allocate IOCBs.
This PR fixes this by making use of the freshly introduced `--reserve-io-control-blocks` seastar option, to reserve IOCBs for tool applications. Since the `linux-aio` and `epoll` reactor backends require quite a bit of these, we enable the `io_uring` reactor backend and switch tools to use this backend instead. The `io_uring` reactor backend needs just 2 IOCBs to function, so the reserve of 10 IOCBs set up in this PR is good for running 5 tool applications in parallel, which should be more than enough.

Fixes: https://github.com/scylladb/scylladb/issues/19185

The problem this PR fixes has a manual workaround (and is rare to begin with), no backport needed.

Closes scylladb/scylladb#21527

* github.com:scylladb/scylladb:
  main: configure a reserve IOCB for scylla-nodetool and friends
  configure: enable the io_uring backend
  main: use configure seastar defaults via app_template::seastar_options
2024-12-09 19:22:19 +02:00
Kefu Chai
a9c244ddf7 dist: scylla_io_setup: use raw string to avoid invalid escape sequence
Use raw string literals to prevent syntax warnings when using regular
expressions with backslash-based patterns.

The original code triggered a SyntaxWarning in developer mode (`python3 -Xdev`)
due to unescaped backslash characters in regex patterns like '\s'. While
CPython typically interprets these silently, strict Python parsing modes
raise warnings about potentially unintended escape sequences.

This change adds the `r` prefix to string literals containing regex patterns,
ensuring consistent behavior across different Python runtime configurations
and eliminating unnecessary syntax warning like:

```
/opt/scylladb/scripts/libexec/scylla_io_setup:41: SyntaxWarning: invalid escape sequence '\s'
  pattern = re.compile(_nocomment + r"CPUSET=\s*\"" + _reopt(_cpuset) + _reopt(_smp) + "\s*\"")
```

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21839
2024-12-09 19:18:39 +03:00
Emil Maskovsky
969b396699 gossiper: fix the backward incompatible change
In the cleanup commit a840949ea0
a regression was introduced that caused backward incompatible changes
in the gossiper application state name strings.

In the e486e0f759 the value
`application_state::CDC_STREAMS_TIMESTAMP` was changed to
`application_state::CDC_GENERATION_ID`, but the name string
"CDC_STREAMS_TIMESTAMP" was kept for backward compatibility.

The cleanup commit a840949ea0 however
changed the name string to "CDC_GENERATION_ID" by ommission (not noticing
the difference) which caused backward incompatible change.

There is also another case found of "IGNOR_MSB_BITS" (that has a typo -
missing the "E" in "IGNORE") to "IGNORE_MSB_BITS", which also needs to
be reverted back to keep the backward compatibility.

Fixes: scylladb/scylladb#21811

Closes scylladb/scylladb#21813
2024-12-09 16:46:25 +01:00
Gleb Natapov
ed7ea1dc71 feature_service: fix typo in address_nodes_by_host_ids feature name
Message-ID: <Z1WYaYuQuPP8lNAX@scylladb.com>
2024-12-09 17:27:27 +02:00
Michael Litvak
53224d90be service/qos: increase timeout of internal get_service_levels queries
The function get_service_levels is used to retrieve all service levels
and it is called from multiple different contexts.
Importantly, it is called internally from the context of group0 state reload,
where it should be executed with a long timeout, similarly to other
internal queries, because a failure of this function affects the entire
group0 client, and a longer timeout can be tolerated.
The function is also called in the context of the user command LIST
SERVICE LEVELS, and perhaps other contexts, where a shorter timeout is
preferred.

The commit introduces a function parameter to indicate whether the
context is internal or not. For internal context, a long timeout is
chosen for the query. Otherwise, the timeout is shorter, the same as
before. When the distinction is not important, a default value is
chosen which maintains the same behavior.

The main purpose is to fix the case where the timeout is too short and causes
a failure that propagates and fails the group0 client.

Fixes scylladb/scylladb#20483

Closes scylladb/scylladb#21748
2024-12-09 13:20:32 +01:00
Kefu Chai
6a18db0aea node_ops: switch from boost::join() to std::ranges::join_view()
Replace boost::join() with std::ranges::join_view() as an interim solution
before C++26's std::views::concat becomes available. This change:

- Reduces dependencies on the Boost Ranges library
- Moves closer to standard library implementations
- Improves code maintainability and future compatibility

This is part of ongoing efforts to modernize our codebase and minimize
external dependencies.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21786
2024-12-09 13:46:44 +03:00
Botond Dénes
2491a31f4c docs: cql/ddl.rst: document {min,max}_index_interval
Closes scylladb/scylladb#21795
2024-12-09 13:45:20 +03:00
Emil Maskovsky
8191e57036 treewide: fix annotations reported by GH checks
Clean up the unnecessary includes reported by the GitHub checks that are
polluting the PR diffs.

The "utils/assert.hh" report should be actually fixed by the #21739, but
as the usage of `SEASTAR_ASSERT()` is protected by the `SEASTAR_DEBUG`
check it makes sense to include the header conditionally as well.

Closes scylladb/scylladb#21817
2024-12-09 13:44:12 +03:00
Avi Kivity
f744007e13 dht: auto_refreshing_sharder.hh: don't include database.hh
database.hh is a heavyweight include file with a lot of fan-in.
auto_refreshing_sharder.hh has a lot of fan out. The combination
means a large dependency load.

Deinline the class and use forward declarations to avoid the #include.

There is no expected performance impact because all the functions are
virtual.

Ref #1

Note: this shouldn't belong in dht, but be injected by a higher layer,
but this isn't addressed by the patch.

Closes scylladb/scylladb#21768
2024-12-06 23:11:52 +01:00
Tomasz Grabiec
7e2875d648 Merge 'Add tablet merge support' from Raphael Raph Carvalho
The goal of merge is to reduce the tablet count for a shrinking table. Similar to how split increases the count while the table is growing. The load balancer decision to merge is implemented today (came with infrastructure introduced for split), but it wasn't handled until now.

Initial tablet count is respected while the table is in "growing mode". For example, the table leaves it if there was a need to split above the initial tablet count. After the table leaves the mode, the average size can be trusted to determine that the table is shrinking. Merge decision is emitted if the average tablet size is 50% of the target. Hysteresis is applied to avoid oscillations between split and merges.

Similar to split, the decision to merge is recorded in tablet map's resize_type field with the string "merge". This is important in case of coordinator failover, so new coordinator continues from where the old left off.

Unlike split, the preparation phase during merge is not done by the replica (with split compactions), but rather by the coordinator by co-locating sibling tablets in the same node's shard. We can define sibling tablets as tablets that have contiguous range and will become one after merge. The concept is based on the power-of-two constraint and token contiguity. For example, in a table with 4 tablets, tablets of ids 0 and 1 are siblings, 2 and 3 are also siblings.

The algorithm for co-locating sibling tablets is very simple. The balancer is responsible for it, and it will emit migrations so that "odd" tablet will follow the "even" one. For example, tablet 1 will be migrated to where tablet 0 lives. Co-location is low in priority, it's not the end of the world to delay merge, but it's not ideal to delay e.g. decommission or even regular load balancing as that can translate into temporary unbalancing, impacting the user activities. So co-location migrations will happen when there is no more important work to do.
While regular balancing is higher in priority, it will not undo the co-location work done so far. It does that by treating co-located tablets as if they were already merged. The load inversion convergence check was adjusted so balancer understand when two tablets are being migrated instead of one, to avoid oscillations.

When balancer completes co-location work for a table undergoing merge, it will put the id of the table into the resize_plan, which is about communicating with the topology coordinator that a table is ready for it. With all sibling tablets co-located, the coordinator can resize the tablet map (reduce it by a factor of 2) and record the new map into group0. All the replicas will react to it (on token metadata update) by merging the storage (memtable(s) + sstables) of sibling tablets into one.

Fixes #18181.

system test details:

test: https://github.com/pehala/scylla-cluster-tests/blob/tablets_split_merge/tablets_split_merge_test.py
yaml file: https://github.com/pehala/scylla-cluster-tests/blob/tablets_split_merge/test-cases/features/tablets/tablets-split-merge-test.yaml

instance type: i3.8xlarge
nodes: 3
target tablet size: 0.5G (scaled down by 10, to make it easier to trigger splits and merges)
description: multiple cycles of growing and shrinking the data set in order to trigger splits and merges.
data_set_size: ~100G
initial_tablets: 64, so it grew to 128 tablets on split, and back to 64 on merge.

latency of reads and writes that happened in parallel to split and merge:
```
$ for i in scylla-bench*; do cat $i | grep "Mode\|99th:\|99\.9th:"; done
Mode:			 write
  99.9th:	 3.145727ms
  99th:		 1.998847ms
  99.9th:	 3.145727ms
  99th:		 2.031615ms
Mode:			 read
  99.9th:	 3.145727ms
  99th:		 2.031615ms
  99.9th:	 3.145727ms
  99th:		 2.031615ms
Mode:			 write
  99.9th:	 3.047423ms
  99th:		 1.933311ms
  99.9th:	 3.047423ms
  99th:		 1.933311ms
Mode:			 read
  99.9th:	 3.145727ms
  99th:		 1.900543ms
  99.9th:	 3.145727ms
  99th:		 1.900543ms
Mode:			 write
  99.9th:	 5.079039ms
  99th:		 3.604479ms
  99.9th:	 35.389439ms
  99th:		 25.624575ms
Mode:			 write
  99.9th:	 3.047423ms
  99th:		 1.998847ms
  99.9th:	 3.047423ms
  99th:		 1.998847ms
Mode:			 read
  99.9th:	 3.080191ms
  99th:		 2.031615ms
  99.9th:	 3.112959ms
  99th:		 2.031615ms
```

Closes scylladb/scylladb#20572

* github.com:scylladb/scylladb:
  docs: Document tablet merging
  tests/boost: Add test to verify correctness of balancer decisions during merge
  tests/topology_experimental_raft: Add tablet merge test
  service: Handle exception when retrying split
  service: Co-locate sibling tablets for a table undergoing merge
  gms: Add cluster feature for tablet merge
  service: Make merge of resize plan commutative
  replica: Implement merging of compaction groups on merge completion
  replica: Handle tablet merge completion
  service: Implement tablet map resize for merge
  locator: Introduce merge_tablet_info()
  service: Rename topology::transition_state::tablet_split_finalization
  service: Respect initial_tablet_count if table is in growing mode
  service: Wire migration_tablet_set into the load balancer
  locator: Add tablet_map::sibling_tablets()
  service: Introduce sorted_replicas_for_tablet_load()
  locator/tablets: Extend tablet_replica equality comparator to three-way
  service: Introduce alias to per-table candidate map type
  service: Add replication constraint check variant for migration_tablet_set
  service: Add convergence check variant for migration_tablet_set
  service: Add migration helpers for migration_tablet_set
  service/tablet_allocator: Introduce migration_tablet_set
  service: Introduce migration_plan::add(migrations_vector)
  locator/tablets: Introduce tablet_map::for_each_sibling_tablets()
  locator/tablets: Introduce tablet_map::needs_merge()
  locator/tablets: Introduce resize_decision::initial_decision()
  locator/tablets: Fix return type of three-way comparison operators
  service: Extract update of node load on migrations
  service: Extract converge check for intra-node migration
  service: Extract erase of tablet replicas from candidate list
  scripts/tablet-mon: Allow visualization of tablet id
2024-12-06 18:06:20 +01:00
Lakshmi Narayanan Sreethar
401e7c8f69 replica/table: fix indent in compaction_group_for_sstable
Signed-off-by: Lakshmi Narayanan Sreethar <lakshmi.sreethar@scylladb.com>
2024-12-06 21:22:24 +05:30
Lakshmi Narayanan Sreethar
fa10b0b390 replica/table: improve error message when encountering orphaned sstables
On startup, if a server reads an sstable that belongs to a tablet that
doesn't have any local replica, it throws an error in the following
format and refuses to start :

```
Storage wasn't found for tablet 1 of table test.test
```

This patch updates the code path to throw a nicer error that includes
the sstable name that caused the problem.

This patch also adds a testcase to verify the error being thrown.

Fixes #18038
2024-12-06 21:22:24 +05:30
Kefu Chai
37c49acbac docs/cql/ddl: Clarify crc_check_chance option behavior
Although `crc_check_chance` is accepted as a configuration option in ScyllaDB,
the value is currently ignored during runtime. This change makes this behavior
explicit in the documentation to prevent potential user misunderstandings.

Changes:
- Explicitly document that the option is currently a no-op
- Provide clear guidance on the current implementation
- Prevent confusion about the option's actual functionality

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21794
2024-12-06 13:48:03 +02:00
Abhinav
6c90a25014 Fix gossiper orphan node floating problem by adding a remover fiber
In the current scenario, if during startup, a node crashes after initiating gossip and before joining group0,
then it keeps floating in the gossiper forever because the raft based gossiper purging logic is only effective
once node joins group0. This orphan node hinders the successor node from same ip to join cluster since it collides
with it during gossiper shadow round.

This commit intends to fix this issue by adding a background thread which periodically checks for such orphan entries in
gossiper and removes them.

A test is also added in to verify this logic. This test fails without this background thread enabled, hence
verifying the behavior.

Fixes: scylladb/scylladb#20082

Closes scylladb/scylladb#21600
2024-12-06 10:45:07 +01:00
Kefu Chai
e04aca7efe github: do not nest ${{}} inside condition
In commit 2596d157, we added a condition to run auto-backport.py only
when the GitHub Action is triggered by a push to the default branch.
However, this introduced an unexpected error due to incorrect condition
handling.

Problem:
- `github.event.before` evaluates to an empty string
- GitHub Actions' single-pass expression evaluation system causes
  the step to always execute, regardless of `github.event_name`

Despite GitHub's documentation suggesting that ${{ }} can be omitted,
it recommends using explicit ${{}} expressions for compound conditions.

Changes:
- Use explicit ${{}} expression for compound conditions
- Avoid string interpolation in conditional statements

Root Cause:
The previous implementation failed because of how GitHub Actions
evaluates conditional expressions, leading to an unintended script
execution and a 404 error when attempting to compare commits.

Example Error:

```
  python .github/scripts/auto-backport.py --repo scylladb/scylladb --base-branch refs/heads/master --commits ..2b07d93beac7bc83d955dadc20ccc307f13f20b6
  shell: /usr/bin/bash -e {0}
  env:
    DEFAULT_BRANCH: master
    GITHUB_TOKEN: ***
Traceback (most recent call last):
  File "/home/runner/work/scylladb/scylladb/.github/scripts/auto-backport.py", line 201, in <module>
    main()
  File "/home/runner/work/scylladb/scylladb/.github/scripts/auto-backport.py", line 162, in main
    commits = repo.compare(start_commit, end_commit).commits
  File "/usr/lib/python3/dist-packages/github/Repository.py", line 888, in compare
    headers, data = self._requester.requestJsonAndCheck(
  File "/usr/lib/python3/dist-packages/github/Requester.py", line 353, in requestJsonAndCheck
    return self.__check(
  File "/usr/lib/python3/dist-packages/github/Requester.py", line 378, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.UnknownObjectException: 404 {"message": "Not Found", "documentation_url": "https://docs.github.com/rest/commits/commits#compare-two-commits", "status": "404"}
```

Fixes scylladb/scylladb#21808
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21809
2024-12-06 11:11:20 +02:00
Kefu Chai
9f5e2488dd locator,service: correct the misspellings
these misspellings were identified by codespell. in this change,
they are corrected.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21796
2024-12-06 11:10:51 +02:00
Piotr Dulikowski
def51e252d Merge 'service/topology_coordinator: migrate view builder only if all nodes are up' from Michał Jadwiszczak
The migration process is doing read with consistency level ALL,
requiring all nodes to be alive.

Fixes scylladb/scylladb#20754

The PR should be backported to 6.2, this version has view builder on group0.

Closes scylladb/scylladb#21708

* github.com:scylladb/scylladb:
  test/topology_custom/test_view_build_status: add reproducer
  service/topology_coordinator: migrate view builder only if all nodes are up
2024-12-06 09:07:07 +01:00
Piotr Dulikowski
c601f7a359 Merge 'transport/server: revert using async function in for_each_gently()' from Michał Jadwiszczak
This patch reverts 324b3c43c0 and adds synchronous versions of `service_level_controller::find_effective_service_level()` and `client_state::maybe_update_per_service_level_params()`.

It isn't safe to do asynchronous calls in `for_each_gently`, as the
connection may be disconnected while a call in callback preempts.

Fixes scylladb/scylladb#21801

Closes scylladb/scylladb#21761

* github.com:scylladb/scylladb:
  Revert "generic_server: use async function in `for_each_gently()`"
  transport/server: use synchronous calls in `for_each_gently` callback
  service/client_state: add synchronous method to update service level params
  qos/service_level_controller: add `find_cached_effective_service_level`
2024-12-06 08:48:41 +01:00
Emil Maskovsky
2b07d93bea raft: clean up the documentation
Small adjustments and improvements to the documentation in the raft
section.

Fixing Markdown lint warnings:
- MD004/ul-style: Unordered list style [Expected: dash; Actual: asterisk]
- MD007/ul-indent: Unordered list indentation [Expected: 0; Actual: 2]
- MD032/blanks-around-lists: Lists should be surrounded by blank lines
- MD036/no-emphasis-as-heading: Emphasis used instead of a heading
- MD046/code-block-style: Code block style [Expected: fenced; Actual: indented]

Closes scylladb/scylladb#21780
2024-12-05 13:44:11 +01:00
Gleb Natapov
636006f976 topology coordinator: do not for replaced node to appear in the gossiper
There is no point waiting for a node been replaced to appear in the
gossiper since it either will be there already or it will never appear.
gossiper:is_alive() knows how to handle both of those cases, so just
call it directly.
2024-12-05 13:36:52 +01:00
Michał Jadwiszczak
fe67efda5b Revert "generic_server: use async function in for_each_gently()"
This reverts commit 324b3c43c0.

It isn't safe to do asynchronous calls in `for_each_gently`, as the
connection may be disconnected while a call in callback preempts.

Fixes scylladb/scylla#21801
2024-12-05 13:32:47 +01:00
Avi Kivity
9024e4940c counters.hh: drop unused boost includes
Re-add them to source files that need them.

Closes scylladb/scylladb#21738
2024-12-05 12:27:41 +02:00
Nadav Har'El
86a8ca8a9f Merge 'Alternator add WCU for delelte item' from Amnon Heiman
This series adds WCU support for the delete item operation.
It also splits the Alternator WCU metric by an ops label to give us better visibility of how much each ops contributes to the WCU calculation.

No need to backport to the open source

Closes scylladb/scylladb#21709

* github.com:scylladb/scylladb:
  test_returnconsumedcapacity.py: Add delete Item tests
  alternator/executor: Add WCU support for delete item
  alternator/executer use uint in describe_item
  alternator/consumed_capacity.hh: Make the total_bytes public
  test_metrics validate split wcu_total to ops
  Alternato: split WCU metrics into ops
2024-12-05 11:27:20 +02:00
Yaron Kaikov
816a8eafbc ./github/workflow/add-label-when-promoted: fix indentaion which preventing the workflow to be triggered when label was added
this workflow should be triggered either if a push event occurred or
pull_request_target (which mean someone added backport label)

It seems that due to wrong indentation the workflow wasn't trigger
during label add

Fixing it

Closes scylladb/scylladb#21791
2024-12-05 09:47:58 +02:00
Pavel Emelyanov
dd8f56ad3a test: Move test_query_built_indexes_virtual_table from boost to cqlpy
And split it into two -- one for materialized view, another for
secondary index. This is to fit current cqlpy layout that has different
files for views and indexes.

refs: #21552
refs: #21551 (detached this patch from there, as that PR needs fix in
the core code)

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>

Closes scylladb/scylladb#21677
2024-12-05 09:17:23 +02:00
Raphael S. Carvalho
d93a0040e5 docs: Document tablet merging
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-12-04 13:11:11 -03:00
Raphael S. Carvalho
8344722a26 tests/boost: Add test to verify correctness of balancer decisions during merge
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-12-04 13:11:11 -03:00
Raphael S. Carvalho
76ab293505 tests/topology_experimental_raft: Add tablet merge test
Passed ./test.py --mode=dev ... --repeat=50.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-12-04 13:11:07 -03:00
Pavel Emelyanov
e1db35c100 Merge 'auth/passwords: Clean up prefix_for_scheme()' from Dawid Mędrek
In this PR, we get rid of the unnecessary default switch case in `prefix_for_scheme()`. We also change the return type of the function to `std::string_view` as it's easier to operate on.

Backport: not needed; this is a code cleanup.

Closes scylladb/scylladb#21749

* github.com:scylladb/scylladb:
  auth/passwords: Change return type of prefix_for_scheme to std::string_view
  auth/passwords.cc: Remove default case in prefix_for_scheme()
2024-12-04 18:38:14 +03:00
Kefu Chai
61ae4a1c86 mutation: remove unused "#include"s
This commit follows up on commit f436edfa22, which initially cleaned up
unused #include directives in the "mutation" subdirectory. This change
removes additional unused header files that were missed in the previous
cleanup.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21740
2024-12-04 15:36:33 +03:00
Kefu Chai
04acf8b075 .github: Add differential-shellcheck workflow for shell script analysis
Introduce a new GitHub workflow to run shellcheck on changed shell
scripts. This workflow automatically detect and highlight potential
shell script issues in pull requests. This change is a follow-up to
commit 0700b322 which fixed an undefined variable issue in `install.sh`.
It intends to leverage static analysis to improve script quality and
catch potential errors early.

Shellcheck will now:
- Analyze all shell scripts modified in pull requests
- Provide inline comments with specific issue details
- Help prevent similar variable-related mistakes in the future

See also
https://github.com/redhat-plumbers-in-action/differential-shellcheck

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21755
2024-12-04 13:34:53 +02:00
Botond Dénes
f55dc71c3f Merge 'Use checksummed input streams in validate_checksums()' from Nikos Dragazis
With commits ed7d352e7d and bb1867c7c7, we now have input streams for both compressed and uncompressed SSTables that provide seamless checksum and digest checking. The code for these was based on `validate_checksums()`, which implements its own validation logic over raw streams. This has led to some duplicate code.

This PR deduplicates the uncompressed case by modifying `validate_checksums()` to use a checksummed input stream instead of a raw stream. The same cannot be done for compressed SSTables though. The reason is that `validate_checksums()` needs to examine the whole data file, even if an invalid chunk is encountered. In the checksummed case we support that by offloading the error handling logic from the data source via a function parameter. In the compressed data source we cannot do that because it needs to return decompressed data and decompression may fail if the data are invalid.

This PR also enables `validate_checksums()` to partially verify SSTables with just the per-chunk checksums if the digest is missing.

In more detail, this PR consists of:
* Port of some integrity checks from `do_validate_uncompressed()` to the checksummed data source. It should now be able to detect corruption due to truncated or appended chunks (expected number of chunks is retrieved from the CRC component).
* Introduction of `error_handler` parameter in checksummed data source and `data_stream()`.
* Refactoring of `validate_checksums()`. The JSON response of `sstable validate-checksums` was also modified to report a missing digest.
*  Tests for `validate_checksums()` against SSTables with truncated data, appended data, invalid digests, or no digest.

Refs #19058.

This PR is a hybrid of cleanup and feature. No backport is needed.

Closes scylladb/scylladb#20933

* github.com:scylladb/scylladb:
  tools/scylla-sstable: Rename valid_checksums -> valid
  test: Check validate_checksums() with missing digest
  sstables: Allow validate_checksums() to report missing digests
  sstables: Refactor validate_checksums() to use checksummed data stream
  sstables: Add error_handler parameter to data_stream()
  sstables: Add error handler in checksummed data source
  sstables: Check for excessive chunks in checksummed data source
  sstables: Check for premature EOF in checksummed data source
  test: test_validate_checksums: Check SSTable with invalid digest
  test: test_validate_checksums: Check SSTable with appended data
  test: test_validate_checksums: Complement test for truncated SSTable
2024-12-04 10:46:18 +02:00
Benny Halevy
d5d4307a20 scylla-sstable: dump-summary: print also first and last tokens
To help scylla-manager restore to map sstables to
nodes or tablets, print also the tokens of the
sstable first and last keys.

For example, the json output will now look like this:
```
$ build/dev/scylla sstable dump-summary /tmp/scylla-344593/data/ks/t-52a92590afd011ef9b68ba86378ed63b/me-3glp_0tm9_00uv52doobo0bvk2t7-big-Data.db | jq
{
  "sstables": {
    "/tmp/scylla-344593/data/ks/t-52a92590afd011ef9b68ba86378ed63b/me-3glp_0tm9_00uv52doobo0bvk2t7-big-Data.db": {
      "header": {
        "min_index_interval": 128,
        "size": 1,
        "memory_size": 16,
        "sampling_level": 128,
        "size_at_full_sampling": 0
      },
      "positions": [
        4
      ],
      "entries": [
        {
          "key": {
            "token": "2008715943680221220",
            "raw": "000400000064",
            "value": "100"
          },
          "position": 0
        }
      ],
      "first_key": {
        "token": "2008715943680221220",
        "raw": "000400000064",
        "value": "100"
      },
      "last_key": {
        "token": "9010454139840013625",
        "raw": "000400000003",
        "value": "3"
      }
    }
  }
}
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes scylladb/scylladb#21735
2024-12-04 10:16:13 +02:00
Botond Dénes
ed08709e75 main: configure a reserve IOCB for scylla-nodetool and friends
Make use of the recently introduced reserve_io_control_blocks to ensure
some reserve IOCBs are left for scylla-nodetool or any other native tool
that might be running intermittently next to ScyllaDB.
These tool apps use the io_uring reactor backend, which requires just 2
IOCBs to function, so the configured default reserve of 10 is good for
running 5 instances of these tools next to ScyllaDB, which should be
good enough.
2024-12-04 02:56:14 -05:00
Botond Dénes
ca956c0180 configure: enable the io_uring backend
To be used by the tool apps -- also change the backend selected in
tools::utils::configure_tool_mode().
We keep using the more mature AIO backend in ScyllaDB itself, so main.cc
sets the linux_aio backend as the default one (the user can still change
this, same as before).
2024-12-04 02:55:31 -05:00
Botond Dénes
f7d66a436e main: use configure seastar defaults via app_template::seastar_options
Instead of the legacy app_template::config. This allows for greater
flexibility, as any option's default can be changed this way, not just
those few that are promoted to app_template::config. This will be made
use of in the next patches.
2024-12-04 02:35:56 -05:00
Raphael S. Carvalho
534ce7340f service: Handle exception when retrying split
It might happen sleep will fail during shutdown, so we should handle
failure for shutdown to proceed gracefully.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-12-03 23:55:44 -03:00
Raphael S. Carvalho
3e518c7b23 service: Co-locate sibling tablets for a table undergoing merge
This implements the ability for the balancer to co-locate sibling
tablets on the same shard. Co-location is low in priority, so
regular load balancer is preferred over it. Previous changes
allowed balancer to move co-located sibling tablets together,
to not undo the co-location work done so far.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-12-03 23:55:43 -03:00
Raphael S. Carvalho
cd5d1d3c99 gms: Add cluster feature for tablet merge
The reason we need it is that tablet merge can only be finalized
when the cluster agrees on the feature, otherwise unpatched
nodes would fail to handle merge finalization, potentially
crashing.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-12-03 21:47:01 -03:00
Raphael S. Carvalho
0a6d41305a service: Make merge of resize plan commutative
set_resize_plan() breaks commutativity since it may override the
resize plans done earlier, for example, when adding co-location
migrations in the DC plan.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-12-03 21:47:01 -03:00
Raphael S. Carvalho
70b3963b8d replica: Implement merging of compaction groups on merge completion
When handling merge completion, compaction groups that belonged to
sibling tablets are placed into the same storage group, since those
tablets become one after merge.

In order to merge two groups, the source group needs its memtable to
be flushed first, such that all the data can be moved into the
destination.

The handling happens in update_effective_replication_map() which cannot
afford to wait for I/O, so the group merge will happen in background.
There's a fiber that will wake up on merge completion and will iterate
through the new set of storage groups (after merge), and will work
on merging additional compaction groups into the main one.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-12-03 21:47:01 -03:00
Raphael S. Carvalho
907739f3d1 replica: Handle tablet merge completion
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-12-03 21:47:01 -03:00
Raphael S. Carvalho
48dcefbf45 service: Implement tablet map resize for merge
This implements the ability to resize the tablet map for merge if
the balancer emits the decision to finalize the merge when all
sibling replicas are colocated for a table. But the co-location
plan is not implemented in the balancer yet, so this is still
not in use.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-12-03 21:47:01 -03:00
Raphael S. Carvalho
014e1c9a0f locator: Introduce merge_tablet_info()
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2024-12-03 21:47:00 -03:00