Rename host_id map getter, 'get_endpoint_to_host_id_map_for_reading' to 'get_endpoint_to_host_id_map_'
Also modify the getter to return information regarding joining nodes as well.
This getter will later be used for retrieving the nodes in nodetool status, hence it needs to show all nodes,
including joining ones.
The function name suffix `_for_reading` suggests that the function was used
in some other places in the past, and indeed if we need endpoints
"for reading" then we cannot show joining endpoints. But it was confirmed
that this function is currently only used by "/storage_service/host_id" endpoint,
hence it can be modified as required.
Fixes: scylladb/scylladb#17857
Collection stress tests include testing of B- B+- and radix trees, and those tests live in unit/ suite. There are also small corner-case tests for those collections in boost/ suite. There's an attempt to get rid of unit suite in favor of boost one, and this PR moves the collections stress testing from unit suite into their boost counterparts.
refs: scylladb/qa-tasks#1655Closesscylladb/scylladb#20475
* github.com:scylladb/scylladb:
test: Move other collection-testing headers from unit to boost
test: Move stress-collecton header from unit to boost
test: Move B+tree compactiont test from unit to boost
test: Move radix tree compactiont test from unit to boost
test: Move B-tree compactiont test from unit to boost
test: Move radix tree stress test from unit to boost
test: Move B-tree stress test from unit to boost
test: Move b+tree stress test from unit to boost
test: Add bool in_thread argument to stress_collection function
The datadir keeps path to directory where local sstables can be. The very same information is now kept in table's storage options (#20542). This set fixes the remaining places that still use table::config::datadir and table::dir() and removes the datadir field.
Closesscylladb/scylladb#20675
* github.com:scylladb/scylladb:
treewide: Remove table::config::datadir
distributed_loader: Print storage options, not datadir
data_dictionary: Add formatter for storage_options
test: Construct table_for_tests with table storage options
test: Generalize pair of make_table_for_tests helpers
tests: Add helper to get snapshot directory from storage options
table: snapshot_exists: Get directory from storage options
table: snapshot_on_all_shards: Get directory from storage options
For each new node added to the raft config populate its ID to IP mapping
in raft address map from the gossiper. The mapping may have expired if a
node is added to the raft configuration long after it first appears in
the gossiper.
Fixesscylladb/scylladb#20600
Backport to all supported versions since the bug may cause bootstrapping failure.
Closesscylladb/scylladb#20601
* github.com:scylladb/scylladb:
test: extend existing test to check that a joining node can map addresses of all pre-existing nodes during join
group0: make sure that address map has an entry for each new node in the raft configuration
After commit d16ea0af, compiling the server using cmake fails with the
following error :
```
FAILED: service/CMakeFiles/service.dir/Dev/qos/service_level_controller.cc.o
...
/home/Scylla/scylladb/cql3/util.hh:21:10: fatal error: 'cql3/CqlParser.hpp' file not found
21 | #include "cql3/CqlParser.hpp"
| ^~~~~~~~~~~~~~~~~~~~
1 error generated.
```
Fix it by linking the cql3 to the service library.
Closesscylladb/scylladb#20805
Maintainers use scripts/pull_github_pr.sh from scylladb.git when merging PRs and before pushing to the next. We want to prevent merges from piling up on top of unstable builds. This change will check Gating's current status and notify the maintainers
Related to scylladb/scylla-pkg#3644Closesscylladb/scylladb#20742
When users create a table using the Alternator API, they can decide if the billing is PROVISIONED of PAY_PER_REQUEST.
If the billing is set to PROVISIONED, they need to set the ProvisionedThroughput ReadCapacityUnits (RCU) and WriteCapacityUnits (WCU).
This series adds support for getting and setting the ProvisionedThroughput. The values will be stored as table extension tags.
Following how TTL is stored within the Alternator, we will use ```system:rcu_attribute``` and ```system:wcu_attribute``` for the labels.
The series adds a test that sets ProvisionedThroughput and validates that it gets the value back. It was tested with both Alternator and AWS.
This series is part of the effort to monitor, limit, and bill Alternator operations.
New code, no need to backport.
Closesscylladb/scylladb#20056
* github.com:scylladb/scylladb:
docs/alternator/compatibility.md: explain the consumed capacity provisioned
Add test/alternator/test_provisioned_throughput.py
test/alternator/util.py: Allow override BillingMode
alternator/executor.cc: Store ProvisionedThroughput
A global index has a primary key of the form
(indexed_column, token, partition_key_column..., clustering_key_column...)
The primary key columns are used to point at the base table row, and
the token (computed as token(partition_key_column...) is used to maintain
sort order.
The query planner has an optimization: if the partition key is fully
constrained to a unique value, then we compute the token from the partition
key and use that to seek directly into the clustering row range for
that base table partition. If the clustering key is also partially
constrained, it is used to refine the index clustering key.
Currently, this optimization is implemented as a hack: the partition key
is extracted from the prepared statement + query options in
get_global_index_token_clustering_ranges(), then used to calculate
the token, which is then substituted in the expression passed to
get_single_column_clustering_bounds() (the expression is shared across
all running queries, so this is quite dangerous).
We simplify the whole thing:
- Let prepare_index_global() recognize that if the partition key is not
fully constrained, then there is no way that we'll be able to compute
the token (as it needs all partition key columns). Since the token
is the first clustering key column of the index table, we can truncate
it to length zero and bail out.
- Otherwise, the partition key is fully constrained. We refactor the
predicate (pk1 = :a AND pk2 = :b) to (pk1, pk2) := (:a, :b). We then
pass expressions representing the partition key to the token function,
ending up with token(:a, :b). We then substitute this expression into
(*_idx_tbl_ck_prefix)[0], which computes the first clustering key
column for the index table.
- Remove the runtime component in get_global_index_clustering_ranges().
Note this include the early return if the partition key wasn't fully
constrained (though the comment only mentions over-constraining), and
the token computation, which is now done by evaluate().
Closesscylladb/scylladb#20733
Before 17f4a151ce the node was marked as
been replaced in join_group0 state, before it actually joins the group0,
so by the time it actually joins and starts transferring snapshot/log no
traffic is sent to it. The commit changed this to mark the node as
being replaced after the snapshot/log is already transferred so we can
get the traffic to the node while it sill did not caught up with a
leader and this may causes problems since the state is not complete.
Mark the node as being replaced earlier, but still add the new node to
the topology later as the commit above intended.
Fixes: scylladb/scylladb#20629
Need to be backported since this is a regression
Closesscylladb/scylladb#20743
* github.com:scylladb/scylladb:
test: amend test_replace_reuse_ip test to check that there is no stale writes after snapshot transfer starts
topology coordinator:: mark node as being replaced earlier
topology coordinator: do metadata barrier before calling finish_accepting_node() during replace
What it called "leader" is actually the destination of the RPC.
Trivial fix, should be backported to all affected versions.
Closesscylladb/scylladb#20789
before this change, `config_file::set_value()` and
`config_file::set_value_on_all_shards()` provide default value for
`config_source`. but the default value is never used -- we alway
specify the `source_source` when calling `set_value_on_all_shards()`.
so in hope to improve the readability, the default value is removed.
so, for example, one can figure out when `config_source::Internal` is
used with less efforts. despite that `config_file::set_value()` is not
used in the tree. for the sake of completeness, its default value is
also dropped.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#20728
with this parameter, "backup" API can backup the given table, this
enables it to be a drop-in replacement of existing rclone API used by
scylla manager.
Fixes https://github.com/scylladb/scylladb/issues/20636
---
this change is a part of the efforts to bring the native backup/restore to scylla, no need to backprt.
Closesscylladb/scylladb#20661
* github.com:scylladb/scylladb:
backup_task: fix the indent
treewide: add "table" parameter to "backup" API
We found that --clang-build-mode INSTALL_FROM tries to rebuild clang
even we use an archive of prebuilt image.
Seems like it is because ninja detected changes on standard library
headers, which updated when we build new frozen toolchain container
image.
To avoid such unnecessary rebuild, we should stop archive whole clang
build directory, we should archive install image instead.
To do so, we can use
"DESTDIR=<sysroot dir> ninja install-distribution-stripped", and archive
sysroot dir as clang archive.
Fixes#20421Closesscylladb/scylladb#20422
This commit modifies the Features page in the following way:
- It adds a short introduction and descriptions to each listed feature.
- It hides the ToC (required to control and modify the information on the page,
e.g., to add descriptions, have full control over what is displayed, etc.)
- Removes the info about Enterprise features (following the request not to include
Enterprise info in the OSS docs)
Fixes https://github.com/scylladb/scylladb/issues/20617
Blocks https://github.com/scylladb/scylla-enterprise/pull/4711Closesscylladb/scylladb#20635
Currently, node ops tasks type is retrieved from topology_request
without any change. Use respective node operation name instead.
Closesscylladb/scylladb#20671
The test performs consecutive schema changes in RECOVERY mode. The
second change relies on the first. However the driver might route the
changes to different servers and we don't have group 0 to guarantee
linearizability. We must rely on the first change coordinator to push
the schema mutations to other servers before returning, but that only
happens when it sees other servers as alive when doing the schema
change. It wasn't guaranteed in the test. Fix this.
Fixesscylladb/scylladb#20791
Should be backported to all branches containing this test to reduce
flakiness.
Closesscylladb/scylladb#20792
with this parameter, "backup" API can backup the given table, this
enables it to be a drop-in replacement of existing rclone API used by
scylla manager.
in this change:
* api/storage_service: add "table" parameter to "backup" API.
* snapshot_ctl: compose the full path of the snapshot directory in
`snapshot_ctl::start_backup`. since we have all the information
for composing the snapshot directory, and what the `backup_task_impl`
class is interested is but the snapshot directory, we just pass
the path to it instead the individual components of the directory.
* backup_task_impl: instead of scan the whole keyspace recursively,
only scan the specified snapshot directory.
Fixesscylladb/scylladb#20636
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Auth has been managed via Raft since Scylla 6.0. Restoring data
following the usual procedure (1) is error-prone and so a safer
method must have been designed and implemented. That's what
happens in this PR.
We want to extend `DESC SCHEMA` by auth and service levels
to provide a safe way to backup and restore those two components.
To realize that, we change the meaning of `DESC SCHEMA WITH INTERNALS`
and add a new "tier": `DESC SCHEMA WITH INTERNALS AND PASSWORDS`.
* `DESC SCHEMA` -- no change, i.e. the statement describes the current
schema items such as keyspaces, tables, views, UDTs, etc.
* `DESC SCHEMA WITH INTERNALS` -- does the same as the previous tier
and also describes auth and service levels. No information about
passwords is returned.
* `DESC SCHEMA WITH INTERNALS AND PASSWORDS` -- does the same
as the previous tier and also includes information about the salted
hashes corresponding to the passwords of roles.
To restore existing roles, we extend the `CREATE ROLE` statement
by allowing to use the option `WITH SALTED HASH = '[...]'`.
---
Implementation strategy:
* Add missing things/adjust existing ones that will be used later.
* Implement creating a role with salted hash.
* Add tests for creating a role with salted hash.
* Prepare for implementing describe functionality of auth and service levels.
* Implement describe functionality for elements of auth and service levels.
* Extend the grammar.
* Add tests for describe auth and service levels.
* Add/update documentation.
---
(1): https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/backup-restore/restore.html
In case the link stops working, restoring a schema was realised
by managing raw files on disk.
Fixesscylladb/scylladb#18750Fixesscylladb/scylladb#18751Fixesscylladb/scylladb#20711Closesscylladb/scylladb#20168
* github.com:scylladb/scylladb:
docs: Update user documentation for backup and restore
docs/dev: Add documentation for DESC SCHEMA
test: Add tests for describing auth and service levels
cql3/functions/user_function: Remove newline character before and after UDF body
cql3: Implement DESCRIBE SCHEMA WITH INTERNALS AND PASSWORDS
auth: Implement describing auth
auth/authenticator: Add member functions for querying password hash
service/qos/service_level_controller: Describe service levels
data_dictionary: Remove keyspace_element.hh
treewide: Start using new overloads of describe
treewide: Fix indentation in describe functions
treewide: Return create statement optionally in describe functions
treewide: Add new describe overloads to implementations of data_dictionary::keyspace_element
treewide: Start using schema::ks_name() instead of schema::keyspace_name()
cql3: Refactor `description`
cql3: Move description to dedicated files
test: Add tests for `CREATE ROLE WITH SALTED HASH`
cql3/statements: Restrict CREATE ROLE WITH SALTED HASH
auth: Allow for creating roles with SALTED HASH
types: Introduce a function `cql3_type_name_without_frozen()`
cql3/util: Accept std::string_view rather than const sstring&
To fix a race between split and repair here c1de4859d8, a new sstable
generated during streaming can be split before being attached to the sstable
set. That's to prevent an unsplit sstable from reaching the set after the
tablet map is resized.
So we can think this split is an extension of the sstable writer. A failure
during split means the new sstable won't be added. Also, the duration of split
is also adding to the time erm is held. For example, repair writer will only
release its erm once the split sstable is added into the set.
This single-sstable split is going through run_custom_job(), which serializes
with other maintenance tasks. That was a terrible decision, since the split may
have to wait for ongoing maintenance task to finish, which means holding erm
for longer. Additionally, if split monitor decides to run split on the entire
compaction group, it can cause single-sstable split to be aborted since the
former wants to select all sstables, propagating a failure to the streaming
writer.
That results in new sstable being leaked and may cause problems on restart,
since the underlying tablet may have moved elsewhere or multiple splits may
have happened. We have some fragility today in cleaning up leaked sstables on
streaming failure, but this single-sstable split made it worse since the
failure can happen during normal operation, when there's e.g. no I/O error.
It makes sense to kill run_custom_job() usage, since the single-sstable split
is offline and an extension of sstable writing, therefore it makes no sense to
serialize with maintenance tasks. It must also inherit the sched group of the
process writing the new sstable. The inheritance happens today, but is fragile.
Fixes#20626.
Closesscylladb/scylladb#20737
* github.com:scylladb/scylladb:
tablet: Fix single-sstable split when attaching new unsplit sstables
replica: Fix tablet split execute after restart
In the current scenario, We check if a node being removed is normal
on the node initiating the removenode request. However, we don't have a
similar check on the topology coordinator. The node being removed could be
normal when we initiate the request, but it doesn't have to be normal when
the topology coordinator starts handling the request.
For example, the topology coordinator could have removed this node while handling
another removenode request that was added to the request queue earlier.
This commit intends to fix this issue by adding more checks in the enqueuing phase
and return errors for duplicate requests for node removal.
This PR fixes a bug. Hence we need to backport it.
Fixes: scylladb/scylladb#20271Closesscylladb/scylladb#20500
We update the relevant articles addressing backing-up
and restoring the schema by specifying that the user
performing it must be a superuser. We also update
the required version of cqlsh.
Additionally, we add an article covering the fundamental
information on `DESCRIBE SCHEMA`.
We add documentation for developers addressing
`DESCRIBE SCHEMA`. It covers the following aspects
of it:
* motivation,
* synopsis of the solution,
* implementation of the solution,
as well as a few subsections explaining the details:
* restoring process and its side effects,
* restoring roles with passwords,
* list of statements generated by `DESC SCHEMA`
with examples,
* implementation details.
We add tests verifying the following features work correctly:
* describing auth: roles, role grants, granting permissions on
resources,
* describing service levels: creating them and attaching to roles.
We remove newline characters that are printed before and after
a UDF's body. This way, we want to keep the create statement
as close to what was actually provided as possible. Although
there should be no semantic differences with or without the
newline characters, it's a lot more convenient in testing when
they're not present.
Fixesscylladb/scylladb#20711
When executing `DESC SCHEMA WITH INTERNALS`, Scylla now also returns
statements that can be used to recreate service levels and restore
the state of auth. That encompasses granting roles and permissions
as well as attaching service levels to roles.
If the additional parameter `WITH PASSWORDS` is provided,
the statements corresponding to recreating roles in the system
will also contain the stored salted hashes.
We introduce a function `describe_auth()` in `auth::service`
responsible for producing a sequence of descriptions whose
corresponding CQL statement can be used to restore the state
of auth.
Add a README.md in test/boost, giving a short introduction to what this
directory is and what kind of tests it contains, and how to run individual
tests.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closesscylladb/scylladb#20550
Now all its users are in boost suite. Once moved, the
stress_collection() function no longer runs in seastar thread, and the
in_thread argument is removed while the function is moved.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This time the boost test needs to stop being pure-boost test, since
bptree compaction test case needs to run in seastar thread. Other
collection tests are already such, not bptree_test joins the party.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This test must run in seastar thread, so put it in seastar-thread test
case, fortunately btree test allows that. Just like its stress peer,
this test also has two invocations from suite, so make it two distinct
test cases as well.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This also moves the code, but takes into account the stress test had two
invovations with suite options -- small and large. Inherit both with two
distinct test cases.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Just move the code. And hard-code the "scale" (i.e. -- number of keys
and iterations) from default arguments of the unit test.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This code is going to be shared between seastar thread and boost tests,
temporarily. So not to yield in pure boost test, add the switch. It will
be removed really soon.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This patch doesn't yet change how schema merging works but it prepares the ground for it by simplifying the code and separating merging logic into its own unit.
It consists of:
- minor cleanups of unused code
- moving code into separate file
- simplifying merge_keyspaces code
More detailed explanation in per commit messages.
Relates scylladb/scylladb#19153Closesscylladb/scylladb#19687
* github.com:scylladb/scylladb:
db: schema_applier: simplify merge_keyspaces function
db: schema_applier: remove unnecessary read in merge_keyspaces
db: schema_tables: move scylla specific code into create keyspace function
db: move schema merging code into a separate unit
db: schema_tables: export some schema management functions
replica: remove unused table_selector forward declaration
db: remove unused flush arg from do_merge_schema func
db: remove unused read_arg_values function
`unspecified` workload type is an internal value and it's not exposed to
user via CQL.
Default value for workload type from user's perspective is `NULL`.
Fixesscylladb/scylladb#20780
Maintainers use scripts/pull_github_pr.sh from scylladb.git when merging PRs and before pushing to the next. We want to prevent merges from piling up on top of unstable builds. This change will check Gating's current status and notify the maintainers
Related to https://github.com/scylladb/scylla-pkg/issues/3644Closesscylladb/scylladb#20742
Because it is never such -- the only caller of truncate_blocking()
always knows the timeout it want this method to use.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closesscylladb/scylladb#20620
Change order of functions: firstly remount, then change ownership for
cgroup. It was not failing before because with privileged mode, it will
mount cgroups as RW, but it's better to have this check if behavior will
change.
Closesscylladb/scylladb#20676
