Commit Graph

865 Commits

Author SHA1 Message Date
Avi Kivity
2fbc532e4d Update tools/python3 submodule
* tools/python3 3e833f1...18fa79e (1):
  > reloc: use `--add-rpath` and not `--set-rpath`
2024-06-30 19:31:23 +03:00
Kefu Chai
77d2d5821d build: cmake: do not mark cqlsh noarch
in 3c7af287, cqlsh's reloc package was marked as "noarch", and its
filename was updated accordingly in `configure.py`, so let's update
the CMake building system accordingly.

this change should address the build failure of

```
08:48:14  [3325/4124] Generating ../Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz
08:48:14  FAILED: Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz /jenkins/workspace/scylla-master/scylla-ci/scylla/build/Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz
08:48:14  cd /jenkins/workspace/scylla-master/scylla-ci/scylla/build/dist && /usr/bin/cmake -E copy /jenkins/workspace/scylla-master/scylla-ci/scylla/tools/cqlsh/build/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz /jenkins/workspace/scylla-master/scylla-ci/scylla/build/Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz
08:48:14  Error copying file "/jenkins/workspace/scylla-master/scylla-ci/scylla/tools/cqlsh/build/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz" to "/jenkins/workspace/scylla-master/scylla-ci/scylla/build/Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz".
```

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#19544
2024-06-30 19:26:54 +03:00
Kefu Chai
947e28146d dbuild: pass --tty when running in interactive mode
podman does not allocate a tty by default, so without `-t` or `--tty`,
one cannot use a functional terminal when interacting with the
container. that what one can expect when running `dbuild -i --`, and
we are greeted with :

```
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
```

after this change, one can enjoy the good-old terminal as usual
after being dropped to the container provided by `dbuild -i --`.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#19550
2024-06-30 12:06:55 +03:00
Avi Kivity
0d23b8165e build: update frozen toolchain to Fedora 40 with clang 18.1.6
This refreshes our dependencies to a supported distribution.

Closes scylladb/scylladb#19205
2024-06-27 14:27:21 +03:00
Israel Fruchter
3c7af28725 cqlsh: update cqlsh submodule
this change updates the cqlsh submodule:

* tools/cqlsh/ ba83aea3...73bdbeb0 (4):
  > install.sh: replace tab with spaces
  > define the the debug packge is empty
  > tests: switch from using cqlsh bash to the test the python file
  > package python driver as wheels

it also includes follow change to package cqlsh as a regular
rpm instead of as a "noarch" rpm:

so far cqlsh bundles the python-driver in, but only as source.
meaning the package wasn't architecture, and also didn't
have the libev eventloop compiled in.

Since from python 3.12 and up, that would mean we would
fallback into asyncio eventloop (which still exprimental)
or into error (once we'll sync with the driver upstream)

so to avoid those, we are change the packaging of cqlsh
to be architecture specific, and get cqlsh compiled, and bundle
all of it's requirements as per architecture installed bundle of wheels.
using `shiv`, i.e. one file virtualenv that we'll be packing
into our artifacts

Ref: https://github.com/scylladb/scylla-cqlsh/issues/90
Ref: https://github.com/scylladb/scylla-cqlsh/pull/91
Ref: https://github.com/linkedin/shiv

Closes scylladb/scylladb#19385

* tools/cqlsh ba83aea...242876c (1):
  > Merge 'package python driver as wheels' from Israel Fruchter

Update tools/cqlsh/ submodule

in which, the change of `define the the debug packge is empty`
should address the build failure like

```
Processing files: scylla-cqlsh-debugsource-6.1.0~dev-0.20240624.c7748f60c0bc.aarch64
error: Empty %files file /jenkins/workspace/scylla-master/next/scylla/tools/cqlsh/build/redhat/BUILD/scylla-cqlsh/debugsourcefiles.list
RPM build errors:
    Empty %files file /jenkins/workspace/scylla-master/next/scylla/tools/cqlsh/build/redhat/BUILD/scylla-cqlsh/debugsourcefiles.list
```

Closes scylladb/scylladb#19473
2024-06-26 12:07:21 +03:00
Avi Kivity
3cfb0503a9 Update tools/cqlsh submodule for v6.0.21-scylla
* tools/cqlsh 0d58e5c...ba83aea (1):
  > requirements: update scylla-driver
2024-06-21 16:04:21 +03:00
Avi Kivity
fdc1449392 treewide: rename flat_mutation_reader_v2 to mutation_reader
flat_mutation_reader_v2 was introduced in a pair of commits in 2021:

  e3309322c3 "Clone flat_mutation_reader related classes into v2 variants"
  08b5773c12 "Adapt flat_mutation_reader_v2 to the new version of the API"

as a replacement for flat_mutation_reader, using range_tombstone_change
instead of range_tombstone to represent represent range tombstones. See
those commits for more information.

The transition was incremental; the last use of the original
flat_mutation_reader was removed in 2022 in commit

  026f8cc1e7 "db: Use mutation_partition_v2 in mvcc"

In turn, flat_mutation_reader was introduced in 2017 in commit

  748205ca75 "Introduce flat_mutation_reader"

To transition from a mutation_reader that nested rows within
a partition in a separate stream, to a flat reader that streamed
partitions and rows in the same stream.

Here, we reclaim the original name and rename the awkward
flat_mutation_reader_v2 to mutation_reader.

Note that mutation_fragment_v2 remains since we still use the original
for compatibilty, sometimes.

Some notes about the transition:

 - files were also renamed. In one case (flat_mutation_reader_test.cc), the
   rename target already existed, so we rename to
    mutation_reader_another_test.cc.

 - a namespace 'mutation_reader' with two definitions existed (in
   mutation_reader_fwd.hh). Its contents was folded into the mutation_reader
   class. As a result, a few #includes had to be adjusted.

Closes scylladb/scylladb#19356
2024-06-21 07:12:06 +03:00
Botond Dénes
495f7160da Update tools/jmx submodule
* tools/jmx 53696b13...3328a229 (1):
  > scylla-apiclient: add missing license for SBOM report
2024-06-18 11:11:57 +03:00
Andrei Chekun
3c921d5712 Add allure pytest adaptor to the toolchain
Add allure-pytest pip dependency to be able to use it for generating the allure report later.
Main benefits of the allure report:
1. Group test failures
2. Possibility to attach log files to she test itself
3. Timeline of test run
4. Test description on the report
5. Search by test name or tag

[avi: regenerate toolchain]

Closes scylladb/scylladb#19335
2024-06-17 23:17:01 +03:00
Yaron Kaikov
996be2e235 dbuild: update toolchain to get latest scylla-api-client
a new Scylla-api-client was released to get a proper license information
in our SBOM report,

Refs: https://github.com/scylladb/scylla-jmx/issues/237

Closes scylladb/scylladb#19324
2024-06-17 08:37:49 +03:00
Kefu Chai
5c41073e00 tools/scylla-sstable: format error message with compile-time check
before this change, we use runtime format string to format error
messages. but it does not have the compile time format check.
if we pass arguments which are not formattable, {fmt} throws at
runtime, instead of error out at compile-time. this could be very
annoying, because we format error messages at the error handling
path. but if user ends up seeing an exception for {fmt} instead
of a nice error message, it would be far from helpful.

in this change, we

- use compile-time format string
- fix two caller sites, where we pass `std::exception_ptr` to
  {fmt}, but `std::exception_ptr` is not formattable by {fmt} at
  the time of writing. we do have operator<< based formatter for
  it though. so we delegate to `fmt::streamed` to format it.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#19294
2024-06-14 15:30:19 +03:00
Botond Dénes
145a67f77c tools/scylla-sstable: log loaded schema with trace level
The schema of the sstable can be interesting, so log it with trace
level. Unfortunately, this is not the nice CQL statement we are used to
(that requires a database object), but the not-nearly-so-nice CFMetadata
printout. Still, it is better then nothing.
2024-06-13 01:32:17 -04:00
Botond Dénes
43c44f0af5 tools/scylla-sstable: load schema from the sstable as fallback
When auto-detecting the schema of the sstable, if all other methods
failed, load the schema from the sstable's serialization header. This
schema is incomplete. It is just enough to parse and display the content
of the sstable. Although parsing and displaying the content of the
sstable is all scylla-sstable does, it is more future-compatible to us
the full schema when possible. So the always-available but minimal
schema that each sstable has on itself, is used just as a fallback.

The test which tested the case when all schema load attempts fail,
doesn't work now, because loading the serialization header always
succeeds. So convert this test into two positive tests, testing the
serialization header schema fallback instead.
2024-06-13 01:32:17 -04:00
Botond Dénes
8f2ba03465 tools/schema_loader: introduce load_schema_from_sstable()
Allows loading the schema from an sstable's serialization header. This
schema is incomplete, but it is enough to parse and display the content
of the sstable.
2024-06-13 01:32:17 -04:00
Avi Kivity
03e776ce3e Update tools/java submodule
* tools/java 88809606c8...01ba3c196f (3):
  > Revert "build: don't add nonexistent directory 'lib' to relocatable packages"
  > build: run antlr in a separate process
  > build: don't add nonexistent directory 'lib' to relocatable packages
2024-06-11 11:58:56 +03:00
Israel Fruchter
1fd600999b Update tools/cqlsh submodule v6.0.20
* tools/cqlsh c8158555...0d58e5ce (6):
  > cqlsh.py: fix server side describe after login command
  > cqlsh: try server-side DESCRIBE, then client-side
  > Refactor tests to accept both client and server side describe
  > github actions: support testing with enterprise release
  > Add the tab-completion support of SERVICE_LEVEL statements
  > reloc/build_reloc.sh: don't use `--no-build-isolation`

Closes scylladb/scylladb#18990
2024-06-06 07:32:05 +03:00
Marcin Maliszkiewicz
2ab143fb40 db: auth: move auth tables to system keyspace
Separate keyspace which also behaves as system brings
little benefit while creating some compatibility problems
like schema digest mismatch during rollback. So we decided
to move auth tables into system keyspace.

Fixes https://github.com/scylladb/scylladb/issues/18098

Closes scylladb/scylladb#18769
2024-05-26 22:30:42 +03:00
Kefu Chai
222dbf2ce4 test/boost: include test/lib/test_utils.hh
this change was created in the same spirit of 505900f18f. because
we are deprecating the operator<< for vector and unorderd_map in
Seastar, some tests do not compile anymore if we disable these
operators. so to be prepared for the change disabling them, let's
include test/lib/test_utils.hh for accessing the printer dedicated
for Boost.test. and also '#include <fmt/ranges.h>' when necessary,
because, in order to format the ranges using {fmt}, we need to
use fmt/ranges.h.

Refs #13245

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-05-26 12:32:43 +08:00
Botond Dénes
2f87bfd634 Update tools/java submodule
* tools/java 4ee15fd9...88809606 (2):
  > Update Scylla Java driver to 3.11.5.3.
  > install-dependencies.sh: s/python/python3/

[botond: regenerate toolchain image]

Closes scylladb/scylladb#18790
2024-05-22 11:39:02 +03:00
Avi Kivity
61505d057e Merge 'Sort user-defined types in describe statements' from Michał Jadwiszczak
User-defined types can depend on each other, creating directed acyclic graph.

In order to support restoring schema from `DESC SCHEMA`, UDTs should be
ordered topologically, not alphabetically as it was till now.

This patch changes the way UDTs are ordered in `DESC SCHEMA`/`DESC KEYSPACE <ks>` statements, so the output can be safely copy-pasted to restore the schema.

Fixes #18539

Closes scylladb/scylladb#18302

* github.com:scylladb/scylladb:
  test/cql-pytest/test_describe: add test for UDTs ordering
  cql3/statements/describe_statement: UDTs topological sorting
  cql3/statements/describe_statement: allow to skip alphabetical sorting
  types: add a method to get all referenced user types
  db/cql_type_parser: use generic topological sorting
  db/cql_type_parses: futurize raw_builder::build()
  test/boost: add test for topological sorting
  utils: introduce generic topological sorting algorithm
2024-05-20 16:58:17 +03:00
Avi Kivity
54a82fed6b feature, index: grandfather CORRECT_IDX_TOKEN_IN_SECONDARY_INDEX
This feature corrected how we store the token in secondary indexes. It
was introduced in 7ff72b0ba5 (2020; 4.4) and can now be assumed present
everywhere. Note that we still support indexes created with the old format.
2024-05-18 00:24:11 +03:00
Michał Jadwiszczak
3830f3bd23 db/cql_type_parses: futurize raw_builder::build()
In order to use generic topological sort,
build() method needs to return future.
2024-05-16 13:30:03 +02:00
Avi Kivity
a5fea84d82 Merge 'scylla-nodetool: add tablet support for ring command' from Botond Dénes
Currently, invoking `nodetool ring` on a tablet keyspace fails with an error, because it doesn't pass the required table parameter to `/storage_service/ownership/{keyspace}`. Further to this, the command will currently always output the vnode ring, regardless of the keyspace and table parameter. This series fixes this, adding tablet support to `/storage_service/tokens_endpoint`, which will now return the tablet ring (tablet token -> tablet primary replica mapping) if the new keyspace and table parameters are provided.
`nodetool status` also gets a touch-up, to provide the tablet ring's token count (the tablet count) when invoked with a tablet keyspace and table.

Fixes: #17889
Fixes: #18474

- [x] ** native-nodetool is new functionality, no backport is needed **

Closes scylladb/scylladb#18608

* github.com:scylladb/scylladb:
  test/nodetool: make test pass with cassandra nodetool
  tools/scylla-nodetool: status: fix token count for tablets
  tools/scylla-nodetool: add tablet support to ring command
  api/storage_service: add tablet support for /storage_service/tokens_endpoint
  service/storage_service: introduce get_tablet_to_endpoint_map()
  locator/tablets: introduce the primary replica concept
2024-05-15 16:05:10 +03:00
Pavel Emelyanov
bb1696910c Merge 'scylla-nodetool: make documentation links product and version dependant' from Botond Dénes
Currently, all documentation links that feature anywhere in the help output of scylla-nodetool, are hard-coded to point to the documentation of the latest stable release. As our documentation is version and product (open-source or enterprise) specific, this is not correct. This PR addresses this, by generating documentation links such that they point to the documentation appropriate for the product and version of the scylladb release.

Fixes: https://github.com/scylladb/scylladb/issues/18276

- [x] the native nodetool is a new feature, no backport needed

Closes scylladb/scylladb#18476

* github.com:scylladb/scylladb:
  tools/scylla-nodetool: make doc link version-specific
  release: introduce doc_link()
  build: pass scylla product to release.cc
2024-05-13 18:03:45 +03:00
Botond Dénes
bec4c17db4 tools/scylla-nodetool: status: fix token count for tablets
Currently, the token count column is always based on the vnodes, which
makes no sense for tablet keyspaces. If a tablet keyspace is provided as
the keyspace argument, don't print the vnode token count. If the user
provided a table argument as well, print the tablet count, otherwise
print "?".
2024-05-13 07:09:20 -04:00
Botond Dénes
e82455beab tools/scylla-nodetool: add tablet support to ring command
Add a table parameter. Pass both keyspace and table (when provided) to
the /storage_service/tokens_endpoint API endpoint, so that the returned
(and printed) token ring is that of the table's tablets, not the vnode
ring.
Also pass the table param to the ownership API, which will complain if
this param is missing for a tablet keyspace.
2024-05-13 07:09:20 -04:00
Israel Fruchter
a1a6bd6798 Update tools/cqlsh submodule to v6.0.18
* tools/cqlsh e5f5eafd...c8158555 (11):
  > cqlshlib/sslhandling: fix logic of `ssl_check_hostname`
  > cqlshlib/sslhandling.py: don't use empty userkey/usercert
  > Dockerfile: noninteractive isn't enough for answering yet on apt-get
  > fix cqlsh version print
  > cqlshlib/sslhandling: change `check_hostname` deafult to False
  > Introduce new ssl configuration for disableing check_hostname
  > set the hostname in ssl_options.server_hostname when SSL is used
  > issue-73 Fixed a bug where username and password from the credentials file were ignored.
  > issue-73 Fixed a bug where username and password from the credentials file were ignored.
  > issue-73
  > github actions: update `cibuildwheel==v2.16.5`

Fixes: scylladb/scylladb#18590

Closes scylladb/scylladb#18591
2024-05-13 07:25:10 +03:00
Botond Dénes
c7c4964b1c tools/scylla-nodetool: make doc link version-specific
Generate documentation link, such that they point to the documentation
page, which is appropriate to the current product (open-source or
enterprise) and version. The documentation links are generated by a new
function and the documentation links are injected into the description
of nodetool command via fmt::format().
2024-05-08 09:41:18 -04:00
Kefu Chai
3421e6dcc1 tools/scylla-nodetool: add formatter for char*
in {fmt} version 10.0.0, it has a regression, which dropped the
formatter for `char *`, even it does format `const char*`, as the
latter is convertible to
`fmt::stirng_view`.

and this issue was addressed in 10.1.0 using 616a4937, which adds
the formatter for `Char *` back, where `Char` is a template parameter.

but we do need to print `vector<char*>`, so, to address the build
failure with {fmt} version 10.0.0, which is shipped along with
fedora 39. let's backport this formatter.

Fixes #18503
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18505
2024-05-02 23:25:24 +03:00
Kefu Chai
bd0d246b57 tools/scylla-nodetool: implement the resetlocalschema command
Fixes #18468
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18470
2024-05-01 08:49:11 +03:00
Raphael S. Carvalho
d7a01598ce tools: Make sstable shard-of efficient by loading minimum to compute owners
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>

Closes scylladb/scylladb#18440
2024-04-30 16:10:58 +03:00
Kefu Chai
6a73c911e3 tools: lua_sstable_consumer.cc: be compatible with Lua 5.3's lua_resume()
in Lua 5.3, lua_resume() only accepts three parameters, while in Lua 5.4,
this function accepts four parameters. so in order to be compatible with
Lua 5.3, we should not pass the 4th parameter to this function.
a macro is defined to conditionally pass this parameter based on the
Lua's version.

see https://www.lua.org/manual/5.3/manual.html#lua_resume

Refs 5b5b8b3264
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18450
2024-04-30 09:06:25 +03:00
Takuya ASADA
af0c0ee8af configure.py: revert changing builddir as absolute path
On be3776ec2a, we changed outdir to
absolute path.
This causes "unknown target" error when we build Scylla using the relative
path something like "ninja build/dev/scylla", since the target name
become absolte path.
Revert the change to able to build with the relative path.

Also, change optimized_clang.sh to use relative path for --builddir,
since we reference "../../$builddir/SCYLLA-*-FILE" when we build
submodule, it won't work with absolute path.

Fixes #18321

Closes scylladb/scylladb#18338
2024-04-29 09:35:21 +03:00
Kamil Braun
d8313dda43 Merge 'db: config: move consistent-topology-changes out of experimental and make it the default for new clusters' from Patryk Jędrzejczak
We move consistent cluster management out of experimental and
make it the default for new clusters in 6.0. In code, we make the
`consistent-topology-changes` flag unused and assumed to be true.

In 6.0, the topology upgrade procedure will be manual and
voluntary, so some clusters will still be using the gossip-based
topology even though they support the raft-based topology.
Therefore, we need to continue testing the gossip-based topology.
This is possible by using the `force-gossip-topology-changes` flag
introduced in scylladb/scylladb#18284.

Ref scylladb/scylladb#17802

Closes scylladb/scylladb#18285

* github.com:scylladb/scylladb:
  docs: raft.rst: update after removing consistent-topology-changes
  treewide: fix indentation after the previous patch
  db: config: make consistent-topology-changes unused
  test: lib: single_node_cql_env: restart a node in noninitial run_in_thread calls
  test: test_read_required_hosts: run with force-gossip-topology-changes
  storage_service: join_cluster: replace force_gossip_based_join with force-gossip-topology-changes
  storage_service: join_token_ring: fix finish_setup_after_join calls
2024-04-26 14:45:29 +02:00
Botond Dénes
7cbe5c78b4 install.sh: use the native nodetool directly
* tools/java b810e8b00e...4ee15fd9ea (1):
  > install.sh: don't install nodetool into /usr/bin

Add a bin/nodetool and install it to bin/ in install.sh. This script
simply forwards to scylla nodetool and it is the replacement for the
Java nodetool, which is dropped from the java-tools's install.sh, in the
submodule update also included in this patch.
With this change, we now hardwire the usage of the native nodetool, as
*the* nodetool, with the intermediary nodetool wrapper script removed
from the picture.
Bash completion was copied from the java tools repository and it is now
installed by the scylla package, together with nodetool.

The Java nodetool is still available as as a fall-back, in case the
native nodetool has problems, at the path of
/opt/scylladb/share/cassandra/bin/nodetool.

Testing

I tested upgrades on a DEB and RPM distro: Ubuntu and Fedora.
First I installed scylla-5.4, then I installed the packages for this PR.
On Ubuntu, I had to use dpkg -i --auto-deconfigure, otherwise, dpkg would
refuse to install the new packages because they break the old ones. No
extra flags were required on Fedora.
In both cases, /usr/bin/nodetool was changed from a thunk calling the
Java nodetool (from 5.4) to the native launcher script from this PR.
/opt/scylladb/share/cassandra/bin/nodetool remained in place and still
works after the upgrade.

I also verified that --nonroot installs also work. Nodetool works both
when called with an absolute path, or when ~/scylladb/bin is added to
$PATH.

Fixes: #18226
Fixes: #17412

Closes scylladb/scylladb#18255

[avi: reset submodule to actual hash we ended up with]
2024-04-25 22:52:00 +03:00
Patryk Jędrzejczak
0d428a3857 treewide: fix indentation after the previous patch 2024-04-25 14:33:21 +02:00
Patryk Jędrzejczak
3a34bb18cd db: config: make consistent-topology-changes unused
We make the `consistent-topology-changes` experimental feature
unused and assumed to be true in 6.0. We remove code branches that
executed if `consistent-topology-changes` was disabled.
2024-04-25 14:33:21 +02:00
Aleksandra Martyniuk
58f72f9019 cql3: statements: change default tombstone_gc mode for tablets
Currently, if tombstone_gc mode isn't specified for a table,
then "timeout" is used by default. With tablets, running
"nodetool repair -pr" may miss a tablet if it migrated across
the nodes. Then, if we expire tombstones for ranges that
weren't repaired, we may get data resurrection.

Set default tombstone_gc mode value for DDLs that don't
specify it. It's set to "repair" for tables which use tablets
unless they use local replication strategy or rf = 1.
Otherwise it's set to "timeout".
2024-04-24 10:42:10 +02:00
Kefu Chai
a439ebcfce treewide: include fmt/ranges.h and/or fmt/std.h
before this change, we rely on the default-generated fmt::formatter
created from operator<<, but fmt v10 dropped the default-generated
formatter.

in this change, we include `fmt/ranges.h` and/or `fmt/std.h`
for formatting the container types, like vector, map
optional and variant using {fmt} instead of the homebrew
formatter based on operator<<.
with this change, the changes adding fmt::formatter and
the changes using ostream formatter explicitly, we are
allowed to drop `FMT_DEPRECATED_OSTREAM` macro.

Refs scylladb#13245

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-04-19 22:56:16 +08:00
Kefu Chai
168ade72f8 treewide: replace formatter<std::string_view> with formatter<string_view>
in in {fmt} before v10, it provides the specialization of `fmt::formatter<..>`
for `std::string_view` as well as the specialization of `fmt::formatter<..>`
for `fmt::string_view` which is an implementation builtin in {fmt} for
compatibility of pre-C++17. and this type is used even if the code is
compiled with C++ stadandard greater or equal to C++17. also, before v10,
the `fmt::formatter<std::string_view>::format()` is defined so it accepts
`std::string_view`. after v10, `fmt::formatter<std::string_view>` still
exists, but it is now defined using `format_as()` machinery, so it's
`format()` method does not actually accept `std::string_view`, it
accepts `fmt::string_view`, as the former can be converted to
`fmt::string_view`.

this is why we can inherit from `fmt::formatter<std::string_view>` and
use `formatter<std::string_view>::format(foo, ctx);` to implement the
`format()` method with {fmt} v9, but we cannot do this with {fmt} v10,
and we would have following compilation failure:

```
FAILED: service/CMakeFiles/service.dir/RelWithDebInfo/topology_state_machine.cc.o
/home/kefu/.local/bin/clang++ -DFMT_DEPRECATED_OSTREAM -DFMT_SHARED -DSCYLLA_BUILD_MODE=release -DSEASTAR_API_LEVEL=7 -DSEASTAR_LOGGER_COMPILE_TIME_FMT -DSEASTAR_LOGGER_TYPE_STDOUT -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -DSEASTAR_SSTRING -DXXH_PRIVATE_API -DCMAKE_INTDIR=\"RelWithDebInfo\" -I/home/kefu/dev/scylladb -I/home/kefu/dev/scylladb/build/gen -I/home/kefu/dev/scylladb/seastar/include -I/home/kefu/dev/scylladb/build/seastar/gen/include -I/home/kefu/dev/scylladb/build/seastar/gen/src -ffunction-sections -fdata-sections -O3 -g -gz -std=gnu++20 -fvisibility=hidden -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wimplicit-fallthrough -Wno-c++11-narrowing -Wno-deprecated-copy -Wno-mismatched-tags -Wno-missing-field-initializers -Wno-overloaded-virtual -Wno-unsupported-friend -Wno-enum-constexpr-conversion -Wno-unused-parameter -ffile-prefix-map=/home/kefu/dev/scylladb=. -march=westmere -mllvm -inline-threshold=2500 -fno-slp-vectorize -U_FORTIFY_SOURCE -Werror=unused-result -MD -MT service/CMakeFiles/service.dir/RelWithDebInfo/topology_state_machine.cc.o -MF service/CMakeFiles/service.dir/RelWithDebInfo/topology_state_machine.cc.o.d -o service/CMakeFiles/service.dir/RelWithDebInfo/topology_state_machine.cc.o -c /home/kefu/dev/scylladb/service/topology_state_machine.cc
/home/kefu/dev/scylladb/service/topology_state_machine.cc:254:41: error: no matching member function for call to 'format'
  254 |     return formatter<std::string_view>::format(it->second, ctx);
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
/usr/include/fmt/core.h:2759:22: note: candidate function template not viable: no known conversion from 'seastar::basic_sstring<char, unsigned int, 15>' to 'const fmt::basic_string_view<char>' for 1st argument
 2759 |   FMT_CONSTEXPR auto format(const T& val, FormatContext& ctx) const
      |                      ^      ~~~~~~~~~~~~
```

because the inherited `format()` method actually comes from
`fmt::formatter<fmt::string_view>`. to reduce the confusion, in this
change, we just inherit from `fmt::format<string_view>`, where
`string_view` is actually `fmt::string_view`. this follows
the document at
https://fmt.dev/latest/api.html#formatting-user-defined-types,
and since there is less indirection under the hood -- we do not
use the specialization created by `FMT_FORMAT_AS` which inherit
from `formatter<fmt::string_view>`, hopefully this can improve
the compilation speed a little bit. also, this change addresses
the build failure with {fmt} v10.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18299
2024-04-19 07:44:07 +03:00
Avi Kivity
6e487a49aa Merge 'toolchain: support building an optimized clang' from Takuya ASADA
This is complete version of #12786, since I take over the issue from @mykaul.
Update from original version are:
- Support ARM64 build (disable BOLT for now since it doesn't functioning)
- Changed toolchain settings to get current scylla able to build (support WASM, etc)
- Stop git clone scylladb repo, create git-archive of current scylla directory and import it
-  Update Clang to 17.0.6
- Save entire clang directory for BUILD mode, not just /usr/bin/clang binary
- Implemented INSTALL_PREBUILT mode to install prebuilt image which built in BUILD mode

Note that this patch drops cross-build support of frozen toolchain, since building clang and scylla multiple time in qemu-user-static will very slow, it's not usable.
Instead, we should build the image for each architecture natively.

----

This is a different way attempting to combine building an optimized clang (using LTO, PGO and BOLT, based on compiling ScyllaDB) to dbuild. Per Avi's request, there are 3 options: skip this phase (which is the current default), build it and build + install it to the default path.

Fixes: #10985
Fixes: scylladb/scylla-enterprise#2539

Closes scylladb/scylladb#17196

* github.com:scylladb/scylladb:
  toolchain: support building an optimized clang
  configure.py: add --build-dir option
2024-04-18 19:20:23 +00:00
Nadav Har'El
e78fc75323 Merge 'tools/scylla-nodetool: add doc link for getsstables and sstableinfo commands' from Botond Dénes
Just like all the other commands already have it. These commands didn't have documentation at the point where they were implemented, hence the missing doc link.

The links don't work yet, but they will work once we release 6.0 and the current master documentation is promoted to stable.

Closes scylladb/scylladb#18147

* github.com:scylladb/scylladb:
  tools/scylla-nodetool: fix typo: Fore -> For
  tools/scylla-nodetool: add doc link for getsstables and sstableinfo commands
2024-04-17 15:15:56 +03:00
Pavel Emelyanov
0f70d276d2 tools/scylla-sstable: Use shorter check is unordered_set contains a key
Currentl code counts the number of keys in it just to see if this number
is non-zero. Using .contains() method is better fit here

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>

Closes scylladb/scylladb#18219
2024-04-16 07:14:48 +03:00
Yaniv Kaul
bd34f2fe46 toolchain: support building an optimized clang
This is a different way attempting to combine building an optimized clang (using LTO, PGO and BOLT, based on compiling ScyllaDB) to dbuild. Per Avi's request, there are 3 options: skip this phase (which is the current default), build it and build + install it to the default path.

Fixes: #10985
Fixes: scylladb/scylla-enterprise#2539
2024-04-08 22:53:59 +09:00
Botond Dénes
6771c646c4 tools/scylla-nodetool: fix typo: Fore -> For 2024-04-03 02:16:59 -04:00
Botond Dénes
b6db56286a tools/scylla-nodetool: add doc link for getsstables and sstableinfo commands
Just like all the other commands already have it. These commands didn't
have documentation at the point where they were implemented, hence the
missing doc link.

The links don't work yet, but they will work once we release 6.0 and the
current master documentation is promoted to stable.
2024-04-03 02:16:03 -04:00
Botond Dénes
fd12052c89 Update tools/java/ submodule
* tools/java/ d61296dc...b810e8b0 (1):
  > do not include {dclocal_,}read_repair_chance if not enabled
2024-04-02 10:47:57 +03:00
Nadav Har'El
566223c34a Merge ' tools/scylla-nodetool: repair: abort on first failed repair' from Botond Dénes
When repairing multiple keyspaces, bail out on the first failed keyspace repair, instead of continuing and reporting all failures at the end. This is what Origin does as well.

To be able to test this, a bit of refactoring was needed, to be able to assert that `scylla-nodetool` doesn't make repair requests, beyond the expected ones.

Refs: https://github.com/scylladb/scylla-cluster-tests/issues/7226

Closes scylladb/scylladb#17678

* github.com:scylladb/scylladb:
  tools/scylla-nodetool: repair: abort on first failed repair
  test/nodetool: nodetool(): add check_return_code param
  test/nodetool: nodetool(): return res object instead of just stdout
  test/nodetool: count unexpected requests
2024-03-28 14:02:29 +02:00
Botond Dénes
81bbfae77a tools/scylla-nodetool: implement the checkAndRepairCdcStreams command
Closes scylladb/scylladb#18076
2024-03-28 13:54:37 +02:00
Botond Dénes
f70f04c240 tools/scylla-nodetool: repair: abort on first failed repair
When repairing multiple keyspaces, bail out on the first failed keyspace
repair, instead of continuing and reporting all failures at the end.
This is what Origin does as well.
2024-03-27 05:46:18 -04:00