Commit Graph

43603 Commits

Author SHA1 Message Date
Aleksandra Martyniuk
00cfc49d18 tasks: introduce task_manager::virtual_task
A virtual task is a new kind of task supported by task manager,
which covers cluster-wide operations.

From users' perspective virtual tasks behave similarly
to task_manager::tasks. The API side of virtual tasks will be
covered in the following patches.

Contrary to task_manager::task, virtual task does not update
its fields proactively. Moreover, no object is kept in memory
for each individual virtual task's operation. Instead a service
(or services) is queried on API user's demand to learn about
the status of running operation. Hence the name.

task_manager::virtual_task is responsible for a whole group
of virtual tasks, i.e. for tracking and generating statuses
of all operations of similar type.

To enable tracking of some kind of operations, one needs to
override task_manager::virtual_task::impl and provide implementations
of the methods returning appropriate information about the operations.
task_manager::virtual_task must be kept on shard 0.

Similarly to task_manager::tasks, virtual tasks can have child tasks,
responsible for tracking suboperations' progress. But virtual tasks
cannot have parents - they are always roots in task trees.

Some methods and structs will be implemented in later patches.
2024-07-23 13:35:01 +02:00
Takuya ASADA
c3bea539b6 dist: support nonroot and offline mode for scylla-housekeeping
Introduce support nonroot and offline mode for scylla-housekeeping.

Closes #13084

Closes scylladb/scylladb#13088
2024-07-23 07:57:32 +03:00
Aleksandra Martyniuk
dfe3af40ed test: tasks: adjust tests to new wait_task behavior
After c1b2b8cb2c /task_manager/wait_task/
does not unregister tasks anymore.

Delete the check if the task was unregistered from test_task_manager_wait.
Check task status in drain_module_tasks to ensure that the task
is removed from task manager.

Fixes: #19351.

Closes scylladb/scylladb#19834
2024-07-22 18:24:54 +03:00
Nadav Har'El
9eb47b3ef0 Merge 'config: round-trip boolean configuration variables' from Avi Kivity
When you SELECT a boolean from system.config, it reads as true/false, but this isn't accepted
on UPDATE (instead, we accept 1/0). This is surprising and annoying, so accept true/false in
both directions.

Not a regression, so a backport isn't strictly necessary.

Closes scylladb/scylladb#19792

* github.com:scylladb/scylladb:
  config: specialize from-string conversion for bool
  config: wrap boost::lexical_cast<> when converting from strings
2024-07-22 17:53:02 +03:00
Botond Dénes
d3135db457 Merge 'commitlog: Add optional max lifetime parameter to cl instance' from Calle Wilund
If set, any remaining segment that has data older than this threshold will request flushing, regardless of data pressure. I.e. even a system where nothing happends will after X seconds flush data to free up the commit log.

Related to  #15820

The functionality here is to prevent pathological/test cases where a silent system cannot fully process stuff like compaction, GC etc due to things like CL forcing smaller GC windows etc.

Closes scylladb/scylladb#15971

* github.com:scylladb/scylladb:
  commitlog: Make max data lifetime runtime-configurable
  db::config: Expose commitlog_max_data_lifetime_in_s parameter
  commitlog: Add optional max lifetime parameter to cl instance
2024-07-22 17:21:33 +03:00
Botond Dénes
3ff33e9c70 Update ./tools/java submodule
* ./tools/java dbaf7ba7...0b4accdd (1):
  > cassandra-stress: Make default repl. strategy NetworkTopologyStrategy

Closes scylladb/scylladb#19818
2024-07-22 17:12:09 +03:00
Kamil Braun
8ec90a0e60 docs: extend "forbidden operations" section for Raft-topology upgrade
The Raft-topology upgrade procedure must not be run concurrently with
version upgrade.

Closes scylladb/scylladb#19746
2024-07-22 12:45:38 +03:00
Botond Dénes
591876b44e Merge 'sstables: do not reload components of unlinked sstables' from Lakshmi Narayanan Sreethar
The SSTable is removed from the reclaimed memory tracking logic only
when its object is deleted. However, there is a risk that the Bloom
filter reloader may attempt to reload the SSTable after it has been
unlinked but before the SSTable object is destroyed. Prevent this by
removing the SSTable from the reclaimed list maintained by the manager
as soon as it is unlinked.

The original logic that updated the memory tracking in
`sstables_manager::deactivate()` is left in place as (a) the variables
have to be updated only when the SSTable object is actually deleted, as
the memory used by the filter is not freed as long as the SSTable is
alive, and (b) the `_reclaimed.erase(*sst)` is still useful during
shutdown, for example, when the SSTable is not unlinked but just
destroyed.

Fixes https://github.com/scylladb/scylladb/issues/19722

Closes scylladb/scylladb#19717

* github.com:scylladb/scylladb:
  boost/bloom_filter_test: add testcase to verify unlinked sstables are not reloaded
  sstables: do not reload components of unlinked sstables
  sstables/sstables_manager: introduce on_unlink method
2024-07-22 12:08:25 +03:00
Avi Kivity
358147959e Merge 'keep table directory open for flushing' from Laszlo Ersek
`filesystem_storage` methods frequently call `sync_directory()`, for the sake of flushing (sync'ing) a directory. `sync_directory()` always brackets the sync with open and close, and given that most `sync_directory()` calls target the sstable base directory, those repeated opens and closes are considered wasteful. Rework the `filesystem_storage::_dir` member (from a mere pathname) so that it stand for an `opened_directory` object, which keeps the sstable base directory open, for the purpose of repeated sync'ing.

Resolves #2399.

Closes scylladb/scylladb#19624

* github.com:scylladb/scylladb:
  sstables/storage: synch "dst_dir" more leanly in create_links_common()
  sstables/storage: close previous directory asynchronously upon dir change
  sstables/storage: futurize change_dir_for_test()
  sstables/storage: sync through "opened_directory" in filesystem...::move()
  sstables/storage: sync through "opened_directory" in the "easy" cases
  sstables/storage: introduce "opened_directory" class
2024-07-21 17:07:44 +03:00
Yaron Kaikov
d3cbe04130 .github/mergify.yml: update conf to support 6.1
Modify Mergify configuation to support `6.1` instead of `5.2` which is
EOL

Closes scylladb/scylladb#19810
2024-07-21 17:02:19 +03:00
Łukasz Paszkowski
781eb7517c api/system: add highest_supported_sstable_format path
Current upgrade dtest rely on a ccm node function to
get_highest_supported_sstable_version() that looks for
r'Feature (.*)_SSTABLE_FORMAT is enabled' in the log files.

Starting from scylla-6.0 ME_SSTABLE_FORMAT is enabled by default
and there is no cluster feature for it. Thus get_highest_supported_sstable_version()
returns an empty list resulting in the upgrade tests failures.

This change introduces a seperate API path that returns the highest
supported sstable format (one of la, mc, md, me) by a scylla node.

Fixes scylladb/scylladb#19772

Backports to 6.0 and 6.1 required. The current upgrade test in dtest
checks scylla upgrades up to version 5.4 only. This patch is a
prerequisite to backport the upgrade tests fix in dtest.

Closes scylladb/scylladb#19787
2024-07-21 17:00:19 +03:00
Avi Kivity
36b57f3432 Merge 'token: inline optimizations' from Benny Halevy
This series contains several optimizations for dht::token
around its comparison functions as well as minimum_token and maximum_token definitions,
by moving them inline into dht/token.hh

This results in a nice improvement in perf-simple-query:
```
==> perf-simple-query.pre <== (21c67a5a64)
         throughput: mean=95774.01 standard-deviation=1129.83 median=96243.64 median-absolute-deviation=1090.08 maximum=96864.09 minimum=94471.19
instructions_per_op: mean=41813.68 standard-deviation=16.27 median=41809.29 median-absolute-deviation=7.02 maximum=41841.64 minimum=41799.41
  cpu_cycles_per_op: mean=22383.19 standard-deviation=331.01 median=22254.53 median-absolute-deviation=332.26 maximum=22744.11 minimum=21996.73

==> perf-simple-query.post.0 <== (token: move ordering operator inline)
         throughput: mean=96350.01 standard-deviation=640.10 median=96228.88 median-absolute-deviation=621.45 maximum=96988.16 minimum=95478.51
instructions_per_op: mean=41627.13 standard-deviation=37.55 median=41627.06 median-absolute-deviation=2.43 maximum=41679.44 minimum=41573.31
  cpu_cycles_per_op: mean=22184.65 standard-deviation=151.03 median=22163.05 median-absolute-deviation=120.83 maximum=22348.49 minimum=21967.30

==> perf-simple-query.post.1 <== (token: operator<=>: optimize the common case)
         throughput: mean=96778.29 standard-deviation=1719.34 median=97021.72 median-absolute-deviation=1059.56 maximum=98300.99 minimum=93893.75
instructions_per_op: mean=41590.25 standard-deviation=5.53 median=41589.50 median-absolute-deviation=4.17 maximum=41598.39 minimum=41584.57
  cpu_cycles_per_op: mean=22135.33 standard-deviation=471.98 median=21969.30 median-absolute-deviation=244.89 maximum=22905.24 minimum=21685.33

==> perf-simple-query.post.3 <== (token: always initialize data member)
         throughput: mean=98264.33 standard-deviation=998.49 median=98533.02 median-absolute-deviation=780.45 maximum=99075.40 minimum=96656.51
instructions_per_op: mean=41657.61 standard-deviation=22.53 median=41648.49 median-absolute-deviation=12.89 maximum=41696.81 minimum=41642.07
  cpu_cycles_per_op: mean=21808.57 standard-deviation=93.63 median=21794.56 median-absolute-deviation=75.41 maximum=21949.46 minimum=21719.55

==> perf-simple-query.post.4 <== (token: constexpr ctors, methods, and minimum/maximum_token)
         throughput: mean=98095.05 standard-deviation=1333.32 median=98930.22 median-absolute-deviation=906.80 maximum=99209.38 minimum=96194.25
instructions_per_op: mean=41572.28 standard-deviation=6.04 median=41574.49 median-absolute-deviation=4.76 maximum=41579.56 minimum=41564.72
  cpu_cycles_per_op: mean=21831.35 standard-deviation=169.56 median=21732.86 median-absolute-deviation=102.93 maximum=22091.66 minimum=21689.63

==> perf-simple-query.post.5 <== (token: initialize non-key tokens with min() value)
         throughput: mean=99502.32 standard-deviation=1003.70 median=99744.03 median-absolute-deviation=388.87 maximum=100482.95 minimum=97813.42
instructions_per_op: mean=41593.48 standard-deviation=17.27 median=41585.25 median-absolute-deviation=8.46 maximum=41619.41 minimum=41575.86
  cpu_cycles_per_op: mean=21545.90 standard-deviation=86.66 median=21578.01 median-absolute-deviation=43.17 maximum=21612.41 minimum=21395.42
```

Optimization only. No backport required

Closes scylladb/scylladb#19782

* github.com:scylladb/scylladb:
  token: initialize non-key tokens with min() value
  token: make kind-based ctor private
  token: constexpr ctors, methods, and minimum/maximum_token
  token: always initialize data member
  everywhere: use dht::token is_{minimum,maximum}
  token: operator<=>: optimize the common case
  token: move ordering operator inline
  partitioner_test: add more token-level tests
2024-07-21 15:07:36 +03:00
Benny Halevy
365e1fb1b9 token: initialize non-key tokens with min() value
We already have code to return min() for
the minimum and maximum tokens in long_token()
and raw(), so instead of using code to return
it, just make sure to set it in the _data member.

Note that although this change affect serialization,
the existing codebase ignores the deserialized bytes
and places a constant (0 before this patch, or min()
with it) in _data for non-key (minumum or maximum) tokens.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-07-20 21:21:42 +03:00
Benny Halevy
9f05072527 token: make kind-based ctor private
Users outside of the token module don't
need to mess with the token::kind.
They can only create key tokens.
Never, minimum or maximum tokens, with a particular
datya value.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-07-20 21:21:42 +03:00
Benny Halevy
6806112189 token: constexpr ctors, methods, and minimum/maximum_token
sizeof(dht::token) is only 16 bytes and therefore
it can be passed with 2 registers.

There is no sense in defining minimum_token
and maximum_token out of line, returning a token&
to statically allocated values that require memory
access/copy, while the only call sites that needs
to point to the static min/max tokens are in
dht::ring_position_view.
Instead, they can be defined inline as constexpr
functions and return their const values.

Respectively, define token ctors and methods
as constexpr where applicable (and noexcept while at it
where applicable)

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-07-20 21:21:42 +03:00
Benny Halevy
e509ccd184 token: always initialize data member
Make sure to always initalize the _data member
to 0 for non-key (minimum or maximum) tokens.

This allows to simplify the equality operator
that now doesn't need to rely on `operator<=>`

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-07-20 21:21:42 +03:00
Benny Halevy
850f298ccd everywhere: use dht::token is_{minimum,maximum}
The is_minimum/is_maximum predicates are more
efficient than comparing the the m{minimum,maximum}_token
values, respectrively. since the is_* functions
need to check only the token kind.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-07-20 21:21:42 +03:00
Benny Halevy
5a60ba5c5f token: operator<=>: optimize the common case
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-07-20 21:21:42 +03:00
Benny Halevy
adc1d7f68f token: move ordering operator inline
Token comparisons are abundant.
The equality operator is defined inline
in dht/token.hh by calling `t1 <=> t2`,
and so is `tri_compare_raw`, which `operator<=>`
calls in the common path, but `operator<=>` itself
is defined out of line, losing the benefits of inlining.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-07-20 21:21:42 +03:00
Benny Halevy
7e745d31ed partitioner_test: add more token-level tests
Before changing how minimum and maximum
tokens are represented in memory.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-07-20 21:21:37 +03:00
Kamil Braun
ad68a7f799 Merge 'test: raft: fix the flaky test_raft_recovery_stuck' from Emil Maskovsky
Use the rolling restart to avoid spurious driver reconnects.

This can be eventually reverted once the scylladb/python-driver#295 is fixed.

Fixes scylladb/scylladb#19154

Closes scylladb/scylladb#19771

* github.com:scylladb/scylladb:
  test: raft: fix the flaky `test_raft_recovery_stuck`
  test: raft: code cleanup in `test_raft_recovery_stuck`
2024-07-19 19:34:43 +02:00
Piotr Dulikowski
4571262e46 Merge 'Improve constness of functions schema code' from Marcin Maliszkiewicz
In v4 of scylladb/scylladb#19598 the last commit of the patch was replaced but this change missed merge so submitting it in a separate patch.

In the current patch, the original functions class correctly marks methods as const where appropriate, and the instance() method now returns a const object. This ensures protection against accidental modifications, as all changes must go through the change_batch object.

Since the functions_changer class was intended to serve the same purpose, it is now redundant. Therefore, we are reverting the commit that introduced it.

Relates scylladb/scylladb#19153

Closes scylladb/scylladb#19647

* github.com:scylladb/scylladb:
  cql3: functions: replace template with std::function in with_udf_iter()
  cql3: functions: improve functions class constness handling
  Revert "cql3: functions: make modification functions accessible only via batch class"
2024-07-19 19:23:11 +02:00
Emil Maskovsky
9ab25e5cbf test: raft: replace the use of read_barrier work-around
Replaced the old `read_barrier` helper from "test/pylib/util.py"
by the new helper from "test/pylib/rest_client.py" that is calling
the newly introduced direct REST API.

Replaced in all relevant tests and decommissioned the old helper.

Introduced a new helper `get_host_api_address` to retrieve the host API
address - which in come cases can be different from the host address
(e.g. if the RPC address is changed).

Fixes: scylladb/scylladb#19662

Closes scylladb/scylladb#19739
2024-07-19 19:20:44 +02:00
Laszlo Ersek
680403d2cd sstables/storage: synch "dst_dir" more leanly in create_links_common()
filesystem_storage::create_links_common() runs on directories that
generally differ from "_dir", thus, we can't replace its sync_directory()
calls with _dir.sync(). We can still use a common (temporary)
"opened_directory" object for synching "dst_dir" three times, saving two
open and two close operations.

This patch is best viewed with "git show -W".

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
2024-07-19 15:46:31 +02:00
Laszlo Ersek
0057ee2431 sstables/storage: close previous directory asynchronously upon dir change
In "filesystem_storage", change_dir_for_test() and move() replace "_dir"
with "opened_directory(new_dir)" using the move assignment operator.
Consequently, the file descriptor underlying "_dir" is closed
synchronously as a part of object destruction.

Expose the async file::close() function through "opened_directory".
Introduce filesystem_storage::change_dir() as a common async workhorse for
both change_dir_for_test() and move(). In change_dir(), close the old
directory asynchronously.

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
2024-07-19 15:43:19 +02:00
Laszlo Ersek
6711574646 sstables/storage: futurize change_dir_for_test()
Currently change_dir_for_test() is synchronous. Make it return a future,
so that we can use async operations in change_dir_for_test() overrides.

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
2024-07-19 15:43:19 +02:00
Laszlo Ersek
ef446c4da0 sstables/storage: sync through "opened_directory" in filesystem...::move()
Near the end of filesystem_storage::move(), we sync both the old
directory, and the new directory, if "delay_commit" is null. At that
point, the new directory is just "_dir"; call _dir.sync() instead of
sync_directory().

This patch is best viewed with "git show -W".

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
2024-07-19 15:14:46 +02:00
Laszlo Ersek
4d33640481 sstables/storage: sync through "opened_directory" in the "easy" cases
Replace

  sst.sstable_write_io_check(sync_directory, _dir.native())

with

  _dir.sync(sst._write_error_handler)

Also replace the explicit (but still relatively "easy")
open_checked_directory() + flush() + flush() operations in
filesystem_storage::seal() with two _dir.sync() calls.

Because filesystem_storage::create_links_common() is marked "const", we
need to declare "_dir" mutable.

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
2024-07-19 15:14:46 +02:00
Laszlo Ersek
2c01171a4d sstables/storage: introduce "opened_directory" class
"filesystem_storage::_dir" is currently of type "std::filesystem::path".
Introduce a new class called "opened_directory", and change the type of
"_dir" to the new class "opened_directory".

"opened_directory" keeps the directory open, and offers synchronization on
that open directory (i.e., without having to reopen the directory every
time). In subsequent patches, that will be put to use.

The opening and closing of the wrapped directory cannot easily be handled
explicitly in the "filesystem_storage" member functions.

(

  Namely, test::store() and test::rewrite_toc_without_scylla_component()
  -- both in "test/lib/sstable_utils.hh" -- perform "open -> ... -> seal"
  sequences, and such a sequence may be executed repeatedly. For example,
  sstable_directory_shared_sstables_reshard_correctly()
  [test/boost/sstable_directory_test.cc] does just that; it "reopens" the
  "filesystem_storage" object repeatedly.

)

Rather than trying to restrict the order of "filesystem_storage" member
function calls, replace the "opened_directory" object with a new one
whenever the directory pathname is re-set; namely in
filesystem_storage::change_dir_for_test() and filesystem_storage::move().

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
2024-07-19 15:14:46 +02:00
Piotr Dulikowski
204a479e82 Merge 'db/hints: Test manager::too_many_in_flight_hints_for()' from Dawid Mędrek
In 6e79d64, the behavior of `manager::too_many_in_flight_hints_for()`
was accidentally modified. It remained unnoticed for some time
and then fixed. In this commit, we add a test verifying that
the concurrency of hints being written to disk is indeed limited
and the limitations are imposed properly.

Refs scylladb/scylladb#17636
Fixes scylladb/scylladb#17660

Closes scylladb/scylladb#19741

* github.com:scylladb/scylladb:
  db/hints: Verify that Scylla limits the concurrency of written hints
  db/hints: Coroutinize `hint_endpoint_manager::store_hint()`
  db/hints: Move a constant value to the TU it's used in
2024-07-19 13:26:34 +02:00
Lakshmi Narayanan Sreethar
0615c8a46b boost/bloom_filter_test: add testcase to verify unlinked sstables are not reloaded
Signed-off-by: Lakshmi Narayanan Sreethar <lakshmi.sreethar@scylladb.com>
2024-07-19 13:15:57 +05:30
Lakshmi Narayanan Sreethar
31ff69a13c sstables: do not reload components of unlinked sstables
The SSTable is removed from the reclaimed memory tracking logic only
when its object is deleted. However, there is a risk that the Bloom
filter reloader may attempt to reload the SSTable after it has been
unlinked but before the SSTable object is destroyed. Prevent this by
removing the SSTable from the reclaimed list maintained by the manager
as soon as it is unlinked.

The original logic that updated the memory tracking in
`sstables_manager::deactivate()` is left in place as (a) the variables
have to be updated only when the SSTable object is actually deleted, as
the memory used by the filter is not freed as long as the SSTable is
alive, and (b) the `_reclaimed.erase(*sst)` is still useful during
shutdown, for example, when the SSTable is not unlinked but just
destroyed.

Fixes #19722

Signed-off-by: Lakshmi Narayanan Sreethar <lakshmi.sreethar@scylladb.com>
2024-07-19 13:15:57 +05:30
Lakshmi Narayanan Sreethar
dbf22848a8 sstables/sstables_manager: introduce on_unlink method
Added a new method, on_unlink() to the sstable_manager. This method is
now used by the sstable to notify the manager when it has been unlinked,
enabling the manager to update its bookkeeping as required. The
on_unlink method doesn't do anything yet but will be updated by the next
patch.

Signed-off-by: Lakshmi Narayanan Sreethar <lakshmi.sreethar@scylladb.com>
2024-07-19 13:15:55 +05:30
Kefu Chai
c52f49facb build: cmake: do not mark cqlsh noarch
in 3c7af287, cqlsh's reloc package was marked as "noarch", and its
filename was updated accordingly in `configure.py`, so let's update
the CMake building system accordingly.

this change should address the build failure of

```
08:48:14  [3325/4124] Generating ../Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz
08:48:14  FAILED: Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz /jenkins/workspace/scylla-master/scylla-ci/scylla/build/Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz
08:48:14  cd /jenkins/workspace/scylla-master/scylla-ci/scylla/build/dist && /usr/bin/cmake -E copy /jenkins/workspace/scylla-master/scylla-ci/scylla/tools/cqlsh/build/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz /jenkins/workspace/scylla-master/scylla-ci/scylla/build/Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz
08:48:14  Error copying file "/jenkins/workspace/scylla-master/scylla-ci/scylla/tools/cqlsh/build/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz" to "/jenkins/workspace/scylla-master/scylla-ci/scylla/build/Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz".
```

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#19710
2024-07-19 08:00:17 +03:00
Kefu Chai
34bf10050b build: cmake: bump up the minimal required fmt to 10.0.0
in cccec07581, we started using a featured introduced by {fmt} v10.
so we need to bump up the required version in CMake as well.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#19709
2024-07-19 07:58:31 +03:00
Botond Dénes
79567c1c98 scripts/open-coredump.sh: allow complete bypass of S3 server
In some cases, the S3 server will not know about a certain build and
any attempt to open a coredump which was generated by this build will
fail, because the S3 server returns an empty/illegal response.
There is already a bypass for missing package-url in the S3 server
response, but this doesn't help in the case when the response is also
missing other metadata, like build-id and version info.
Extend this existig mechanism with a new --scylla-package-url flag,
which provides complete bypass. When provided, the S3 server will not be
queried at all, instead the package is downloaded from the link and
version metadata is extracted from the package itself.

Closes scylladb/scylladb#19769
2024-07-18 21:43:53 +03:00
Avi Kivity
58a8fd6f19 Update tools/python3 submodule (install umask, selinux)
* tools/python3 18fa79e...fbf12d0 (1):
  > install.sh: fix incorrect permission on strict umask

Ref https://github.com/scylladb/scylladb/issues/8589
Ref https://github.com/scylladb/scylladb/issues/19775
2024-07-18 21:36:50 +03:00
Avi Kivity
7984e595ce Update tools/java submodule (install selinux context)
* tools/java 33938ec16f...dbaf7ba7db (1):
  > install.sh: apply correct security context on offline installer

Ref https://github.com/scylladb/scylladb/issues/8589
2024-07-18 21:03:32 +03:00
Kefu Chai
4fbfecbb3e Update seastar submodule
* seastar 908ccd93...67065040 (44):
  > metrics: Use this_shard_id unconditionally
  > sstring: prevent fmt from formatting sstring as a sequence
  > coding style: allow lines up to 160 chars in length
  > src/core: remove unnecessary includes
  > when_all: stop using deprecated std::aligned_union_t
  > reactor: respect preempt requests in debug mode
  > core: fix -Wunused-but-set-variable
  > gate: add try_hold
  > sstring: declare nested type with typename
  > rpc: pass start time to `wait_for_reply()` which accepts `no_wait_type`
  > scripts/perftune.py: get rid of "SyntaxWarning: invalid escape sequence"
  > scripts/perftune.py: add support for tweaking VLAN interfaces
  > scripts/perftune.py: improve discovery of bond device slaves
  > scripts/perftune.py: refactor __learn_slaves() function
  > code-cleanup: add missing header guards
  > code-cleanup: remove redundant includes of 'reactor.hh'
  > code-cleanup: explicitly depend on io_desc.hh
  > scripts/perftune.py: aRFS should be disabled by default in non-MQ mode
  > code-cleanup: remove unneeded includes of fair_queue.hh
  > docker: fix mount of install-dependencies
  > code-cleanup: remove redundant includes of linux-aio.hh
  > fstream: reformat the doxygen comment of make_file_input_stream()
  > iostream: use new-style consumer to implement copy()
  > stall-analyser: use 0 for default value of --minimum
  > reactor: fix crash during metrics gathering
  > build: run socket test with linux-aio reactor backend
  > test: Add testing of connect()-ion abort ability
  > linux_perf_event: exclude_idle only on x86_64
  > linux_perf_event: add make_linux_perf_event
  > stall-analyser: gracefully handle empty input
  > shared_token_bucket: resolve FIXME
  > io_tester: ensure that file object is valid when closing it
  > tutorial.md: fix typo in Dan Kegel's name
  > test,rpc: Extend simple ping-pong case
  > rpc: Calculate delay and export it via metrics
  > rpc: Exchange handler duration with server responses
  > rpc: Track handler execution time
  > rpc: Fix hard-coded constants when sending unknown verb reply
  > reactor: Unfriend alien and smp queues
  > reactor: Add and use stopped() getter
  > reactor: Generalize wakeup() callers
  > file: Use lighter access to map of fs-info-s
  > file: Fix indentation after previous patch
  > file: Don't return chain of ready futures from make_file_impl

Closes scylladb/scylladb#19780
2024-07-18 20:00:15 +03:00
Avi Kivity
f7e24cf0b1 Update tools/jmx submodule (umask fix)
* tools/jmx 3328a22...89308b7 (1):
  > install.sh: fix incorrect permission on strict umask

Ref scylladb/scylladb#14383
Ref scylladb/scylladb#8589
2024-07-18 19:37:57 +03:00
Avi Kivity
c3b9e64713 Merge 'sstable::open_sstable: pass origin from the writer' from Lakshmi Narayanan Sreethar
Pass origin when opening the sstable from the writer and store it in the
sstable object. This will make the origin available for the entire write
path.

Closes scylladb/scylladb#19721

* github.com:scylladb/scylladb:
  sstables: use _origin in write path
  sstable::open_sstable: pass and store origin
2024-07-18 19:30:32 +03:00
Avi Kivity
926a02451e Merge 'sstables/index_reader: abort reading during shutdown' from Lakshmi Narayanan Sreethar
This PR adds support for aborting index reads from within `index_consume_entry_context::consume_input` when the server is being stopped. The abort source is now propagated down to the `index_consume_entry_context`, making it available for `consume_input` to check if an abort has been requested. If an abort is detected, `consume_input` will throw an exception to stop the index read operation.

Closes scylladb/scylladb#19453

* github.com:scylladb/scylladb:
  test/boost: test abort behaviour during index read
  sstables/index_reader: stop consuming index when abort has been requested
  sstables::index_consume_entry_context: store abort_source
  sstable: drop old filter only after the new filter is built during rebuild
  sstables/sstables_manager: store abort_source in sstable_manager
  replica/database: pass abort_source to database constructor
2024-07-18 19:26:22 +03:00
Avi Kivity
0780228aa2 config: specialize from-string conversion for bool
The yaml/json representation for bool is true/false, but boost::lexical_cast
is 1/0. Specialize bool conversion to accept true/false (for yaml/json
compatibilty) and 1/0 (for backward compatibility). This provides
round-trip conversion for bool configs in system.config.
2024-07-18 18:38:22 +03:00
Avi Kivity
33eaa61cdd config: wrap boost::lexical_cast<> when converting from strings
Configuration uses boost::lexical_cast to convert strings to native
values (e.g. bools/ints). However, boost::lexical_cast doesn't
recognize true/false for bool. Since we can't change boost::lexical_cast,
replace it with a wrapper that forwards directly to boost::lexical_cast.

In the next step, we'll specialize it for bool.
2024-07-18 18:38:19 +03:00
Anna Mikhlin
cd007123c3 Update ScyllaDB version to: 6.2.0-dev 2024-07-18 16:07:07 +03:00
Avi Kivity
47e99f4e04 Merge 'Fix lwt semaphore guard accounting' from Gleb Natapov
Currently the guard does not account correctly for ongoing operation if semaphore acquisition fails. It may signal a semaphore when it is not held.

Should be backported to all supported versions.

Closes scylladb/scylladb#19699

* github.com:scylladb/scylladb:
  test: add test to check that coordinator lwt semaphore continues functioning after locking failures
  paxos: do not signal semaphore if it was not acquired
2024-07-18 14:58:31 +03:00
Dawid Medrek
8b6e887e02 db/hints: Verify that Scylla limits the concurrency of written hints
In 6e79d64, the behavior of `manager::too_many_in_flight_hints_for()`
was accidentally modified. It remained unnoticed for some time
and then fixed. In this commit, we add a test verifying that
the concurrency of hints being written to disk is indeed limited
and the limitations are imposed properly.
2024-07-18 13:49:29 +02:00
Kefu Chai
db56af2e41 replication_strategy: mark fmt::formatter<..>::format() const
since fmt 11, it is required that the format() to be const, otherwise
its caller in fmt library would not be able to call it. and compile
would fail like:

```
/home/kefu/.local/bin/clang++ -DFMT_SHARED -DSCYLLA_BUILD_MODE=release -DSEASTAR_API_LEVEL=7 -DSEASTAR_LOGGER_COMPILE_TIME_FMT -DSEASTAR_LOGGER_TYPE_STDOUT -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -DSEASTAR_SSTRING -DXXH_PRIVATE_API -DCMAKE_INTDIR=\"RelWithDebInfo\" -I/home/kefu/dev/scylladb -I/home/kefu/dev/scylladb/seastar/include -I/home/kefu/dev/scylladb/build/seastar/gen/include -I/home/kefu/dev/scylladb/build/seastar/gen/src -I/home/kefu/dev/scylladb/build/gen -isystem /home/kefu/dev/scylladb/abseil -ffunction-sections -fdata-sections -O3 -g -gz -std=gnu++23 -fvisibility=hidden -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wimplicit-fallthrough -Wno-c++11-narrowing -Wno-deprecated-copy -Wno-mismatched-tags -Wno-missing-field-initializers -Wno-overloaded-virtual -Wno-unsupported-friend -Wno-enum-constexpr-conversion -Wno-unused-parameter -ffile-prefix-map=/home/kefu/dev/scylladb=. -march=westmere -Xclang -fexperimental-assignment-tracking=disabled -mllvm -inline-threshold=2500 -fno-slp-vectorize -U_FORTIFY_SOURCE -Werror=unused-result -MD -MT locator/CMakeFiles/scylla_locator.dir/RelWithDebInfo/abstract_replication_strategy.cc.o -MF locator/CMakeFiles/scylla_locator.dir/RelWithDebInfo/abstract_replication_strategy.cc.o.d -o locator/CMakeFiles/scylla_locator.dir/RelWithDebInfo/abstract_replication_strategy.cc.o -c /home/kefu/dev/scylladb/locator/abstract_replication_strategy.cc
In file included from /home/kefu/dev/scylladb/locator/abstract_replication_strategy.cc:9:
In file included from /home/kefu/dev/scylladb/locator/abstract_replication_strategy.hh:16:
In file included from /home/kefu/dev/scylladb/gms/inet_address.hh:11:
In file included from /usr/include/fmt/ostream.h:23:
In file included from /usr/include/fmt/chrono.h:23:
In file included from /usr/include/fmt/format.h:41:
/usr/include/fmt/base.h:1393:23: error: no matching member function for call to 'format'
 1393 |     ctx.advance_to(cf.format(*static_cast<qualified_type*>(arg), ctx));
      |                    ~~~^~~~~~
/usr/include/fmt/base.h:1374:21: note: in instantiation of function template specialization 'fmt::detail::value<fmt::context>::format_custom_arg<locator::vnode_effective_replication_map::factory_key, fmt::formatter<locator::vnode_effective_replication_map::factory_key>>' requested here
 1374 |     custom.format = format_custom_arg<
      |                     ^
/home/kefu/dev/scylladb/seastar/include/seastar/util/log.hh:299:33: note: in instantiation of function template specialization 'fmt::format_to<seastar::internal::log_buf::inserter_iterator &, locator::vnode_effective_replication_map::factory_key &, const void *, 0>' requested here
  299 |                     return fmt::format_to(it, fmt.format, std::forward<Args>(args)...);
      |                                 ^
/home/kefu/dev/scylladb/seastar/include/seastar/util/log.hh:428:9: note: in instantiation of function template specialization 'seastar::logger::log<locator::vnode_effective_replication_map::factory_key &, const void *>' requested here
  428 |         log(log_level::debug, std::move(fmt), std::forward<Args>(args)...);
      |         ^
/home/kefu/dev/scylladb/locator/abstract_replication_strategy.cc:561:18: note: in instantiation of function template specialization 'seastar::logger::debug<locator::vnode_effective_replication_map::factory_key &, const void *>' requested here
  561 |         rslogger.debug("create_effective_replication_map: found {} [{}]", key, fmt::ptr(erm.get()));
      |                  ^
/home/kefu/dev/scylladb/locator/abstract_replication_strategy.hh:471:10: note: candidate function template not viable: 'this' argument has type 'const fmt::formatter<locator::vnode_effective_replication_map::factory_key>', but method is not marked const
  471 |     auto format(const locator::vnode_effective_replication_map::factory_key& key, FormatContext& ctx) {
      |          ^
1 error generated.
```

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#19768
2024-07-18 13:52:36 +03:00
Avi Kivity
c93e2662ae build: regenerate toolchain for optimized clang
Generate a profile-guided-optimization build of clang and install it.
See bd34f2fe46.

The optimized clang package can be found in

  https://devpkg.scylladb.com/clang/clang-18.1.6-Fedora-40-x86_64.tar.gz
  https://devpkg.scylladb.com/clang/clang-18.1.6-Fedora-40-aarch64.tar.gz

Closes scylladb/scylladb#19685
2024-07-18 12:57:45 +03:00
Botond Dénes
8cc99973eb Merge 'Apply sstable io error handler to exceptions generated when opening file' from Calle Wilund
Fixes #19753

SSTable file open provides an `io_error_handler` instance which is applied to a file-wrapper to process any IO errors happing during read/write via the handler in `storage_service`, which in turn will effectively disable the node. However, this is not applied to the actual open operation itself, i.e. any exception generated by the file open call itself will instead just escape to caller.

This PR adds filtering via the `error_handler` to sstable open + makes `storage_service` "isolate" mechanism non-module-static (thus making it testable) and adds tests to check we exhibit the same behaviour in both cases.

The main motivation for this issue it discussions that secondary level IO issues (i.e. caused by extensions) should trigger the same behaviour as, for example, running out of disk space.

Closes scylladb/scylladb#19766

* github.com:scylladb/scylladb:
  memtable_test: Add test for isolate behaviour on exceptions during flush
  cql_test_env: Expose storage service
  storage_service: Make isolate guard non-static and add test accessor
  sstable: apply error_handler on open exceptions
2024-07-18 08:14:40 +03:00