before this change, we rely on the default-generated fmt::formatter
created from operator<<, but fmt v10 dropped the default-generated
formatter.
in this change, we include `fmt/ranges.h` and/or `fmt/std.h`
for formatting the container types, like vector, map
optional and variant using {fmt} instead of the homebrew
formatter based on operator<<.
with this change, the changes adding fmt::formatter and
the changes using ostream formatter explicitly, we are
allowed to drop `FMT_DEPRECATED_OSTREAM` macro.
Refs scylladb#13245
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Currently all scylla-tools assume that the operation/command is in
argv[1]. This is not very flexible, because most programs allow global
options (that are not dependent on the current operation/command) to be
passed before the operation name on the command line. Notably C*'s
nodetool is one such program and indeed scripts and tests using nodetool
do utilize this.
This patch makes this more flexible. Instead of looking at argv[1], do
an initial option parsing with boost::program_options to locate the
operation parameter. This initial parser knows about the global options,
and the operation positional argument. It allows for unrecognized
positional and non-positional arguments, but only after the command.
With this, any combination of global options + operation is allowed, in
any order.
Unfortunately, we have code in scylla-nodetool.cc which needs to know
what are the current help options available. Soon, there will be more
code like this in tools/utils.cc, so centralize this list in a const
static tool_app_template member.
The default AIO backend requires AIO blocks. On production systems, all
available AIO blocks could have been already taken by ScyllaDB. Even
though the tools only require a single unit, we have seen cases where
not even that is available, ScyllDB having siphoned all of the available
blocks.
We could try to ensure all deployments have some spare blocks, but it is
just less friction to not have to deal with this problem at all, by just
using the epoll backend. We don't care about performance in the case of
the tools anyway, so long as they are not unreasonably slow. And since
these tools are replacing legacy tools written in Java, the bar is low.
Closesscylladb/scylladb#17438
get0() dates back from the days where Seastar futures carried tuples, and
get0() was a way to get the first (and usually only) element. Now
it's a distraction, and Seastar is likely to deprecate and remove it.
Replace with seastar::future::get(), which does the same thing.
Currently, tool_app_template::run_async() crashes when invoked with
empty argv (with just argv[0] populated). This can happen if the tool
app is invoked without any further args, e.g. just invoking `scylla
nodetool`. The crash happens because unconditional dereferencing of
argv[1] to get the current operation.
To fix, add an early-exit for this case, just printing a usage message
and exiting with exit code 2.
In other words, have all tools pass their name without the "scylla-"
prefix to `tool_app_template::config::name`. E.g., replace
"scylla-nodetool" with just "nodetool".
Patch all usages to re-add the prefix if needed.
The app name is just more flexible this way, some users might want the
name without the "scylla-" prefix (in the next patch).
Use operation_option to describe positional options. The structure used
before -- app_template::positional_option -- was not a good fit for
this, as it was designed to store a description that is immediately
passed to the boost::program_options subsystem and then discarded.
As such, it had a raw pointer member, which was expected to be
immediately wrapped by boost::shared_ptr<> by boost::program_options.
This produced memory leaks for tools, for options that ended up not
being used. To avoid this altogether, use operation_option, converting
to the app_template::positional_option at the last moment.
Currently, the tools loosely follow the following convention on
error-codes:
* return 1 if the error is with any of the command-line arguments
* return 2 on other errors
This patch changes the returned error-code on unknown operation/command
to 100 (instead of the previous 1). The intent is to allow any wrapper
script to determine that the tool failed because the operation is
unrecognized and not because of something else. In particular this
should enable us to write a wrapper script for scylla-nodetool, which
dispatches commands still un-implemented in scylla-nodetool, to the java
nodetool.
Note that the tool will still print an error message on an unknown
operation. So such wrapper script would have to make sure to not let
this bleed-through when it decides to forward the operation.
Closesscylladb/scylladb#15517
Currently, operation-options are declared in a single global list, then
operations refer to the options they support via name. This system was
born at a time, when scylla-sstable had a lot of shared options between
its operations, so it was desirable to declare them centrally and only
add references to individual operations, to reduce duplication.
However, as the dust settled, only 2 options are shared by 2 operations
each. This is a very low benefit. Up to now the cost was also very low
-- shared options meant the same in all operations that used them.
However this is about to change and this system becomes very awkward to
use as soon as multiple operations want to have an option with the same
name, but sligthly (or very) different meaning/semantics.
So this patch changes moves the options to the operations themselves.
Each will declare the list of options it supports, without having to
reference some common list.
This also removes an entire (although very uncommon) class of bugs:
option-name referring to inexistent option.
Closes#14898
instead of using a loop of std::swap(), let's use std::shift_left()
when appropriate. simpler and more readable this way.
moreover, the pattern of looking for a command and consume it from
the command line resembles what we have in main(), so let's use
similar logic to handle both of them. probably we can consolidate
them in future.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#14888
The skeleton of the two existing scylla-native tools (scylla-types and
scylla-sstable) is very similar. By skeleton, I mean all the boilerplate
around creating and configuring a seastar::app_template, representing
operations/command and their options, and presenting and selecting
these.
To facilitate code-sharing and quick development of any new tools,
extract this skeleton from scylla-sstable.cc into tools/utils.hh,
in the form of a new tool_app_template, which wraps a
seastar::app_template and centralizes all the boilerplate logic in a
single place. The extracted code is not a simple copy-paste, although
many elements are simply copied. The original code is not removed yet.
Use the new seastar option to instruct seastar to not initialize and use
the seastar allocator, relying on the standard allocator instead.
Configure LSA with the standard allocator based segment store backend:
* scylla-types reserves 1MB for LSA -- in theory nothing here should use
LSA, but just in case...
* scylla-sstable reserves 100MB for LSA, to avoid excessive trashing in
the sstable index caches.
With this, tools now should allocate memory on demand, without reserving
a large chunk of (or all of) the available memory, as regular seastar
apps do.
Instead of lengthy blurbs, switch to single-line, machine-readable
standardized (https://spdx.dev) license identifiers. The Linux kernel
switched long ago, so there is strong precedent.
Three cases are handled: AGPL-only, Apache-only, and dual licensed.
For the latter case, I chose (AGPL-3.0-or-later and Apache-2.0),
reasoning that our changes are extensive enough to apply our license.
The changes we applied mechanically with a script, except to
licenses/README.md.
Closes#9937
Which configures seastar to act more appropriate to a tool app. I.e.
don't act as if it owns the place, taking over all system resources.
These tools are often run on a developer machine, or even next to a
running scylla instance, we want them to be the least intrusive
possible.
Also use the new tool mode in the existing tools.
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20211220143104.132327-1-bdenes@scylladb.com>
