Although ScyllaDB ignores this request parameter, the Java nodetools
sets it, so it is better to have the native one do the same for
symmetry. It makes testing easier.
Discovered with the more strict request matching introduced in the next
patches.
This PR implements the following new nodetool commands:
* netstats
* tablehistograms/cfhistograms
* proxyhistograms
All commands come with tests and all tests pass with both the new and the current nodetool implementations.
Refs: https://github.com/scylladb/scylladb/issues/15588Closesscylladb/scylladb#17651
* github.com:scylladb/scylladb:
tools/scylla-nodetool: implement the proxyhistograms command
tools/scylla-nodetool: implement the tableshistograms command
tools/scylla-nodetool: introduce buffer_samples
utils/estimated_histogram: estimated_histogram: add constructor taking buckets
tools/scylla-nodetool: implement the netstats command
tools/scylla-nodetool: add correct units to file_size_printer
When printing human-readable file-sizes, the Java nodetool always uses
base-2 steps (1024) to arrive at the human-readable size, but it uses
the base-10 units (MB) and base-2 units (MiB) interchangeably.
Adapt file_size_printer to support both. Add a flag to control which is
used.
Calling scylla-nodetool with option describering and ommiting the keyspace
name argument results in a boost exception with the following error message:
error running operation: boost::wrapexcept<boost::bad_any_cast> (boost::bad_any_cast: failed conversion using boost::any_cast)
This change checks for the missing keyspace and outputs a more sensible
error message:
error processing arguments: keyspace must be specified
Closesscylladb/scylladb#17741
instead of using fmt::runtime format string, use compile-time
format string, so that we can have compile-time format check provided
by {fmt}.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#17709
before this change, "ring" subcommand has two issues:
1. `--resolve-ip` option accepts a boolean argument, but this option
should be a switch, which does not accept any argument at all
2. it always prints the endpoint no matter if `--resolve-ip` is
specified or not. but it should print the resolved name, instead
of an IP address if `--resolve-ip` is specified.
in this change, both issues are addressed. and the test is updated
accordingly to exercise the case where `--resolve-ip` is used.
Closesscylladb/scylladb#17553
* github.com:scylladb/scylladb:
tools/scylla-nodetool: print hostname if --resolve-ip is passed to "ring"
test/nodetool: calc max_width from all_hosts
test/nodetool: keep tokens as Host's member
test/nodetool: remove unused import
* tools/java 5e11ed17...e4878ae7 (2):
> nodetool: fix a typo in error message
> bin/cassandra-stress: Add extended version info
Closesscylladb/scylladb#17680
before this change, if `buildah` is not available in $PATH, this script
fails like:
```console
$ tools/toolchain/prepare --help
tools/toolchain/prepare: line 3: buildah: command not found
```
the error message never gets a chance to show up. as `set -e` in the
shebang line just let bash quit.
after this change, we check for the existence of buildah, and bail out
if it is not available. so, on a machine without buildah around, we now
have:
```console
$ tools/toolchain/prepare --help
install buildah 1.19.3 or later
```
the same applies to "reg".
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#17697
The following incompatibilities were identified by `listsnapshots_test.py` in dtests:
* Command doesn't bail out when there are no snapshots, instead it prints meaningless empty report
* Formatting is incompatible
Both are fixed in this mini-series.
Closesscylladb/scylladb#17541
* github.com:scylladb/scylladb:
tools/scylla-nodetool: listsnapshots: make the formatting compatible with origin's
tools/scylla-nodetool: listsnapshots: bail out if there are no snapshots
before this change, "ring" subcommand has two issues:
1. `--resolve-ip` option accepts a boolean argument, but this option
should be a switch, which does not accept any argument at all
2. it always prints the endpoint no matter if `--resolve-ip` is
specified or not. but it should print the resolved name, instead
of an IP address if `--resolve-ip` is specified.
in this change, both issues are addressed. and the test is updated
accordingly to exercise the case where `--resolve-ip` is used.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
When no keyspace is provided, request all keyspaces from the server,
then scrub all of them. This is what the legacy nodetool does, for some
reason this was missed when re-implementing scrub.
Closesscylladb/scylladb#17495
The author (me) tried to be clever and fix the formatting, but then he
realized this just means a lot of unnecessary fighting with tests. So
this patch makes the formatting compatible with that of the legacy
nodetool:
* Use compatible rounding and precision formatting
* Use incorrect unit (KB instead of KiB)
* Align numbers to the left
* Add trailing white-space to "Snapshot Details: "
These two parameters are not used by the native nodetool, because
ScyllaDB itself doesn't support them. These should be just ignored and
indeed there was a unit test checking that this is the case. However,
due to a mistake in the unit test, this was not actually tested and
nodetool complained when seeing these params.
This patch fixes both the test and the native nodetool.
Closesscylladb/scylladb#17477
* tools/cqlsh b8d86b76...e5f5eafd (2):
> dist/debian: fix the trailer line format
> `COPY TO STDOUT` shouldn't put None where a function is expected
Fixes: scylladb/scylladb#17451Closesscylladb/scylladb#17447
instead of passing fmt string as a plain `const char*`, pass it as
a consteval type, so that `fmt::format()` can perform compile-time
format check against it and the formatted params.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#17656
This patch series makes all auth writes serialized via raft. Reads stay
eventually consistent for performance reasons. To make transition to new
code easier data is stored in a newly created keyspace: system_auth_v2.
Internally the difference is that instead of executing CQL directly for
writes we generate mutations and then announce them via raft group0. Per
commit descriptions provide more implementation details.
Refs https://github.com/scylladb/scylladb/issues/16970
Fixes https://github.com/scylladb/scylladb/issues/11157Closesscylladb/scylladb#16578
* github.com:scylladb/scylladb:
test: extend auth-v2 migration test to catch stale static
test: add auth-v2 migration test
test: add auth-v2 snapshot transfer test
test: auth: add tests for lost quorum and command splitting
test: pylib: disconnect driver before re-connection
test: adjust tests for auth-v2
auth: implement auth-v2 migration
auth: remove static from queries on auth-v2 path
auth: coroutinize functions in password_authenticator
auth: coroutinize functions in standard_role_manager
auth: coroutinize functions in default_authorizer
storage_service: add support for auth-v2 raft snapshots
storage_service: extract getting mutations in raft snapshot to a common function
auth: service: capture string_view by value
alternator: add support for auth-v2
auth: add auth-v2 write paths
auth: add raft_group0_client as dependency
cql3: auth: add a way to create mutations without executing
cql3: run auth DML writes on shard 0 and with raft guard
service: don't loose service_level_controller when bouncing client_state
auth: put system_auth and users consts in legacy namespace
cql3: parametrize keyspace name in auth related statements
auth: parametrize keyspace name in roles metadata helpers
auth: parametrize keyspace name in password_authenticator
auth: parametrize keyspace name in standard_role_manager
auth: remove redundant consts auth::meta::*::qualified_name
auth: parametrize keyspace name in default_authorizer
db: make all system_auth_v2 tables use schema commitlog
db: add system_auth_v2 tables
db: add system_auth_v2 keyspace
When a tool application is invoked with an unknown operation, an error
message is printed, which includes all the known operations, with all
their aliases. This is collected in `std::vector<std::string_view>`. The
problem is that the vector containing alias names, is returned as a
value, so the code ends up creating views to temporaries.
Fix this by returning alias vector with const&.
Fixes: #17584Closesscylladb/scylladb#17586
before this change, we failed to apply the filtering of tablestats
command in the right way:
1. `table_filter` failed to check if delimiter is npos before
extract the cf component from the specified table name.
2. the stats should not included the keyspace which are not
included by the filter.
3. the total number of tables in the stats report should contain
all tables no matter they are filtered or not.
in this change, all the problems above are addressed. and the tests
are updated to cover these use cases.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#17468
- use API endpoint of /storage_service/toppartition/
- only print out the specified samplings.
- print "\n" separator between samplings
Closesscylladb/scylladb#17574
* github.com:scylladb/scylladb:
tools/scylla-nodetool: print separator between samplings
tools/scylla-nodetool: only print the specified sampling
tools/scylla-nodetool: use /storage_service/toppartition/
New keyspace is added similarly as system_schema keyspace,
it's being registred via system_keyspace::make which calls
all_tables to build its schema.
Dummy table 'roles' is added as keyspaces are being currently
registered by walking through their tables. Full table schemas
will be added in subsequent commits.
Change can be observed via cqlsh:
cassandra@cqlsh> describe keyspaces;
system_auth_v2 system_schema system system_distributed_everywhere
system_auth system_distributed system_traces
cassandra@cqlsh> describe keyspace system_auth_v2;
CREATE KEYSPACE system_auth_v2 WITH replication = {'class': 'LocalStrategy'} AND durable_writes = true;
CREATE TABLE system_auth_v2.roles (
role text PRIMARY KEY
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
AND comment = 'comment'
AND compaction = {'class': 'SizeTieredCompactionStrategy'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 604800
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
instead of printing it out after samplings, we should print it
in between them. as toppartitions_test.py in dtest splits the
samplings using "\n\n". without this change, dtest would consider
the empty line as another sampling and then fail the test. as
the empty sampling does not match with the expected regular expressions.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
before this change, we print all samplings returned by the API,
but this is not what cassandra nodetool's behavior, which only
prints out the specified one. and the toppartitions_test.py
in dtest actually expects that the number of sampling should
match with the one specified with command line.
so, in this change, we only print out the specified samplings.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
instead of using the endpoint of /storage_service/toppartition,
use /storage_service/toppartition/. otherwise API server refuses
to return the expected result. as it does match with any API endpoint.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
the "keyspace" argument of the "ring" command is optional. but before
this change, we considered it a mandatory option. it was wrong.
so, in this change, we make it optional, and print out the warning
message if the keyspace is not specified.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#17472
* tighten the param check for toppartitions
* add an extra empty line inbetween reports
Closesscylladb/scylladb#17486
* github.com:scylladb/scylladb:
tools/scylla-nodetool: add an extra empty line inbetween reports
tools/scylla-nodetool: tighten the param check for toppartitions
simpler this way.
Closesscylladb/scylladb#17437
* github.com:scylladb/scylladb:
tools/scylla-nodetool: use {yaml,json}_writers in compactionhistory_operation
tools/scylla-nodetool: add {json,yaml}_writer
before this change, `toppartitions` does not print an empty line
after an empty sampling warning message. but
dtest/toppartitions_test.py actually split sampling reports with
two newlines, so let's appease it. the output also looks better
this way, as the samplings for READS and WRITES are always visually
separated with an empty line.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
the test cases of `test_any_of_required_parameters_is_missing`
considers that we should either pass all positional argument or
pass none of them, otherwise nodetool should fail. but `scylla nodetool`
supported partial positional argument.
to be more consistent with the expected behavior, in this change,
we enforce the sanity check so that we only accept either all
positional args or none of them. the corresponding test is added.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
we already have generic operator<< based formatter for sequence-alike
ranges defined in `utils/to_string.hh`, but as a part of efforts to
address #13245, we will eventually drop the formatter.
to prepare for this change, we should create/find the alternatives
where the operator<< for printing the ranges is still used.
Boost::program_options is one of them. it prints the options' default
values using operator<< in its error message or usage. so in order
to keep it working, we define operator<< for `vector<sstring>` here.
if there are more types are required, we will need the generalize
this formatter. if there are more needs from different compiling
units, we might need to extract this helper into, for instance,
`utils/to_string.hh`. but we should do this after removing it.
Refs #13245
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#17413
instead of materializing the `managed_bytes_view` to a string, and
print it, print it directly to stdout. this change helps to deprecate
`to_hex()` helpers, we should materialize string only when necessary.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#17463
The default AIO backend requires AIO blocks. On production systems, all
available AIO blocks could have been already taken by ScyllaDB. Even
though the tools only require a single unit, we have seen cases where
not even that is available, ScyllDB having siphoned all of the available
blocks.
We could try to ensure all deployments have some spare blocks, but it is
just less friction to not have to deal with this problem at all, by just
using the epoll backend. We don't care about performance in the case of
the tools anyway, so long as they are not unreasonably slow. And since
these tools are replacing legacy tools written in Java, the bar is low.
Closesscylladb/scylladb#17438
so that we have less repeatings for dumping the metrics. the repeatings
are error-prone and not maintainable. also move them out into a separate
header, to keep fit of this source file -- it's now 3000 LOC. also,
by moving them out, we can reuse them in other subcommands without
moving them to the top of this source file.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
As usual, the new command is covered with tests, which pass with both the legacy and the new native implementation.
Refs: #15588Closesscylladb/scylladb#17368
* github.com:scylladb/scylladb:
tools/scylla-nodetool: implement the repair command
test/nodetool: utils: add check_nodetool_fails_with_error_contains()
test/nodetool: util: replace flags with custom matcher
this change introduces a new exception which carries the status code
so that an operation can return a non-zero exit code without printing
any errors. this mimics the behavior of "viewbuildstatus" command of
C* nodetool.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#17359
