Otherwise, rpm dependency resolution starts by installing an older
version of gcc (to satisfy an older preinstalled libgcc dependency),
then updates it. After the change, we install the updated gcc in
the first place.
`prepare` builds a multiarch image using qemu emulation. It turns
out that aarch64 emulation is slowest (due to emulating pointer
authentication) so it makes sense to run it on an aarch64 host. To do
that, we need only to adjust the check for qemu installation.
Unfortunately, docker arch names and Linux arch names are different,
so we have to add an ungainly translation, but otherwise it is a
simple loop.
The main reason for adding rust dependency to scylla is the
wasmtime library, which is written in rust. Although there
exist c++ bindings, they don't expose all of its features,
so we want to do that ourselves using rust's cxx.
Signed-off-by: Wojciech Mitros <wojciech.mitros@scylladb.com>
[avi: update toolchain]
[avi: remove example, saving for a follow-on]
"
Examining sstables of system tables is quite a common task. Having to
dump the schemas of such tables into a schema.cql is annoying knowing
that these schemas are readily available in scylla, as they are
hardcoded. This mini-series adds a method to make use of this fact, by
adding a new option: `--system-schema`, which takes the name of a system
table and looks up its schema.
Tests: unit(dev)
"
* 'scylla-sstable-system-schema/v1' of https://github.com/denesb/scylla:
tools/scylla-sstable: add alternative schema load method for system tables
tools/schema_loader: add load_system_schema()
db/system_distributed_keyspace: add all tables methods
tools/scylla-sstable: reorganize main help text
Currently the main help is a big wall of text. This makes it hard to
quickly jump to the section of interest. This patch reorganizes it into
clear sections, each with a title. Sections are now also ordered
according to the part they reference in the command-line.
This should make it easier for answers to questions regarding a certain
topic to be quickly found, without having to read a lot of text.
In most files it was unused. We should move these to the patch which
moved out the last interesting reader from mutation_reader.hh (and added
the corresponding new header include) but its probably not worth the
effort.
Some other files still relied on mutation_reader.hh to provide reader
concurrency semaphore and some other misc reader related definitions.
And pass it to the cql3 layer when parsing statements. This allows the
schema loader to cut itself from replica::database, using a local, much
simpler database implementation. This not only makes the code much
simpler but also opens up the way to using the standard allocator in
tools. The real database uses LSA which is incompatible with the
standard allocator (in release builds that is).
"
This patchset adds two new operations to scylla-sstable:
* validate-checksums - helps identifying whether an sstable is intact or
not, but checking the digest and the per-chunk checksums against the
data on disk.
* decompress - helps when one wants to manually examine the content of a
compressed sstable.
Refs: #497
Tests: unit(dev)
"
* 'scylla-sstable-validate-checksums-decompress/v3' of https://github.com/denesb/scylla:
tools/scylla-sstable: consume_sstables(): s/no_skips/use_crawling_reader/
tools/scylla-sstable: add decompress operation
tools/scylla-sstables: add validate-checksums operation
sstables/sstable: add validate_checksums()
sstables/sstable: add raw_stream option to data_stream()
sstables/sstable: make data_stream() and data_read() public
utils/exceptions: add maybe_rethrow_exception()
Due to fd62fba985
scoped enums are not automatically converted to integers anymore,
this is the intended behavior, according to the fmtlib devs.
A bit nicer solution would be to use `std::to_underlying`
instead of a direct `static_cast`, but it's not available until
C++23 and some compilers are still missing the support for it.
Tests: unit(dev)
Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
Just like scylla-sstable, have a separate --help content for reach
action. The existing description is shortened and is demoted to summary:
this now only appears in the listing in the main description.
Currently the keyspace is only auto-created for create type statements.
However the keyspace is needed even without UDTs being involved: for
example if the table contains a collection type. So auto-create the
keyspace unconditionally before preparing the first statement.
Also add a test-case with a create table statement which requires the
keyspace to be present at prepare time.
Wrapping a rapidjson::Writer<> and mirrors the latter's API, providing
more convenient overloads for the Key() and String() methods, as well as
providing some extra, scylla-sstable specific methods too.
Extract the actual dumping code into a separate class, which also
implements sstable_consumer interface. The dumping consumer now just
forwards calls to actual dumper through the abstract consumer interface,
allowing different concrete dumpers to be instantiated.
Since ME sstable format includes originating host id in stats
metadata, local host id needs to be made available for writing and
validation.
Both Scylla server (where local host id comes from the `system.local`
table) and unit tests (where it is fabricated) must be accomodated.
Regardless of how the host id is obtained, it is stored in the db
config instance and accessed through `sstables_manager`.
Signed-off-by: Michael Livshin <michael.livshin@scylladb.com>
Instead of lengthy blurbs, switch to single-line, machine-readable
standardized (https://spdx.dev) license identifiers. The Linux kernel
switched long ago, so there is strong precedent.
Three cases are handled: AGPL-only, Apache-only, and dual licensed.
For the latter case, I chose (AGPL-3.0-or-later and Apache-2.0),
reasoning that our changes are extensive enough to apply our license.
The changes we applied mechanically with a script, except to
licenses/README.md.
Closes#9937
"
Currently commands are regular switches. This has several disadvantages:
* CLI programs nowadays use the command-based UX, so our tools are
awkward to use to anybody used to that;
* They don't stand out from regular options;
* They are parsed at the same time as regular options, so all options
have to be dumped to a single description;
This series migrates the tools to the command based CLI. E.g. instead of
scylla sstable --validate --merge /path/to/sst1 /path/to/sst2
we now have:
scylla sstable validate --merge /path/to/sst1 /path/to/sst2
Which makes it much clearer that "validate" is the command and "merge"
is an option. And it just looks better.
Internally the command is parsed and popped from argv manually just as
we do with the tool name in scylla main(). This means we know the
command before even building the
boost::program_options::options_description representation and thus
before creating the seastar::app_template instance. Consequently we can
tailor the options registered and the --help content (the application
description) to the command run.
So now "scylla sstable --help" prints only a general description of the
tool and a list of the supported operations. Invoking "scylla sstable
{operation} --help" will print a detailed description of the operation
along with its specific options. This greatly improves the documentation
and the usability of the tool.
"
Refs #9882
* 'tools-command-oriented-cli/v1' of https://github.com/denesb/scylla:
tools/scylla-sstable: update general description
tools/scylla-sstable: proper operation-specific --help
tools/scylla-sstable: proper operation-specific options
tools/scylla-sstable: s/dump/dump-data/
tools/utils: remove now unused get_selected_operation() overload
tools: take operations (commands) as positional arguments
tools/utils: add positional-argument based overload of get_selected_operation()
tools: remove obsolete FIXMEs
We now have detailed per-operation descriptions, so remove
operation-specific parts of the general one and instead add more details
on the common options and arguments.
Add a detailed description to each of the operations. This description
replaces the general one when the operation specific help is displayed
(scylla sstable {operation} --help). The existing short description of
the operations is demoted to a summary and is made even shorter. This
will serve as the headline on the operation specific help page, as well
as the summary on the operation listing.
This allows the specifics of each operation to be detailed in length
instead of the terse summary that was available before.
Operation-specific options are a mess currently. Some of them are in the
general options, all individual operations having to check for their
presence and warn if unsupported ones are set. These options were
general only when scylla-sstable had a single operation (dump). They
(most of them) became specific as soon as a second one was added.
Other specific options are in the awkward to use (both on the CLI
and in code) operation-specific option map.
This patch cleans this mess up. Each operation declares the option it
supports and these are only added to the command line when the specific
operation is chosen. General options now only contain options that are
truly universal.
As a result scylla-sstable has a operation-specific --help content now.
Operation-specific options are only printed when the operation is
selected:
scylla sstable --help
will only print generic options, while:
scylla sstable dump-data --help
will also print options specific to said operation. The description is
the same still, but this will be fixed in the next patch too.
Move replica-oriented classes to the replica namespace. The main
classes moved are ::database, ::keyspace, and ::table, but a few
ancillary classes are also moved. There are certainly classes that
should be moved but aren't (like distributed_loader) but we have
to start somewhere.
References are adjusted treewide. In many cases, it is obvious that
a call site should not access the replica (but the data_dictionary
instead), but that is left for separate work.
scylla-gdb.py is adjusted to look for both the new and old names.
We now have dump-{component} for all sstable components, so rename dump
to dump-data to follow the established naming scheme and to clear any
possible confusion about what it dumps.
