Update docs for backup procedure to use `DESC SCHEMA WITH INTERNALS`
instead of plain `DESC SCHEMA`.
Add a note to use cqlsh in a proper version (at least 6.0.19).
Closesscylladb/scylladb#18953
This change supports changing replication factor in tablets-enabled keyspaces.
This covers both increasing and decreasing the number of tablets replicas through
first building topology mutations (`alter_keyspace_statement.cc`) and then
tablets/topology/schema mutations (`topology_coordinator.cc`).
For the limitations of the current solution, please see the docs changes attached to this PR.
Fixes: #16129Closesscylladb/scylladb#16723
* github.com:scylladb/scylladb:
test: Do not check tablets mutations on nodes that don't have them
test: Fix the way tablets RF-change test parses mutation_fragments
test/tablets: Unmark RF-changing test with xfail
docs: document ALTER KEYSPACE with tablets
Return response only when tablets are reallocated
cql-pytest: Verify RF is changes by at most 1 when tablets on
cql3/alter_keyspace_statement: Do not allow for change of RF by more than 1
Reject ALTER with 'replication_factor' tag
Implement ALTER tablets KEYSPACE statement support
Parameterize migration_manager::announce by type to allow executing different raft commands
Introduce TABLET_KEYSPACE event to differentiate processing path of a vnode vs tablets ks
Extend system.topology with 3 new columns to store data required to process alter ks global topo req
Allow query_processor to check if global topo queue is empty
Introduce new global topo `keyspace_rf_change` req
New raft cmd for both schema & topo changes
Add storage service to query processor
tablets: tests for adding/removing replicas
tablet_allocator: make load_balancer_stats_manager configurable by name
This commit adds the information that the manual recovery procedure
is not supported if tablets are enabled.
In addition, the content in the Manual Recovery Procedure is reorganized
by adding the Prerequisites and Procedure subsections - in this way,
we can limit the number of Note and Warning boxes that made the page
hard to follow.
Fixes https://github.com/scylladb/scylladb/issues/18895Closesscylladb/scylladb#18935
Because ALTER KS will result in creating a global topo req, we'll have
to pass the req data to topology coordinator's state machine, and the
easiest way to do it is through sytem.topology table, which is going to
be extended with 3 extra columns carrying all the data required to
execute ALTER KS from within topology coordinator.
This commit adds the main description of tablets and their
benefits.
The article can be used as a reference in other places
across the docs where we mention tablets.
Closesscylladb/scylladb#18619
Separate keyspace which also behaves as system brings
little benefit while creating some compatibility problems
like schema digest mismatch during rollback. So we decided
to move auth tables into system keyspace.
Fixes https://github.com/scylladb/scylladb/issues/18098Closesscylladb/scylladb#18769
This doesn't apply for auth-v2 as we improved data placement and
removed cassandra quirk which was setting different CL for some
default superuser involved operations.
Fixes#18773Closesscylladb/scylladb#18785
This commit enables publishing documentation
from branch-6.0. The docs will be published
as UNSTABLE (the warning about version 6.0
being unstable will be displayed).
Closesscylladb/scylladb#18832
This commit fixes the incorrect Raft-related information on the Handling Cluster Membership Change Failures page
introduced with https://github.com/scylladb/scylladb/pull/17500.
The page describes the procedure for when Raft is disabled. Since 6.0, Raft for consistent schema management
is enabled and mandatory (cannot be disabled), this commit adds the procedure for Raft-enabled setups.
Closesscylladb/scylladb#18803
This commit updates the documentation about Raft in version 6.0.
- "Introduction": The outdated information about consistent topology updates not being supported
is removed and replaced with the correct information.
- "Enabling Raft": The relevant information is moved to other sections. The irrelevant information
is removed. The section no longer exists.
- "Verifying that the Raft upgrade procedure finished successfully" - moved under Schema
(in the same document). I additionally removed the include saying that after you verify
that schema on Raft is enabled, you MUST enable topology changes on Raft (it is not mandatory;
also, it should be part of the upgrade guide, not the Raft document).
- Unnecessary or incorrect references to versions are removed.
Refs https://github.com/scylladb/scylladb/issues/18580Closesscylladb/scylladb#18689
This commit replaces the 5.4-to-5.5 upgrade guide with the 5.4-to-6.0 upgrade guide,
including the metrics update information.
The guide references the "Enable Consistent Topology Updates" document,
as enabling consistent topology updates is a new step when upgrading to version 6.0.
Also, a procedure for image upgrades has been added (as verified by @yaronkaikov).
Fixesscylladb/scylladb#18254Fixesscylladb/scylladb#17896
Refs scylladb/scylladb#18580Closesscylladb/scylladb#18728
Update `docs/dev/isolation.d`:
* Update the list of scheduling groups
* Remove IO priority groups (they were folded into scheduling groups)
* Add section on RPC isolation
Closesscylladb/scylladb#18749
* github.com:scylladb/scylladb:
docs: isolation.md: add section on RPC call isolation
docs: isolation.md: remove mention of IO priority groups
docs: isolation.md: update scheduling group list, add aliases
There's a nasty scenario when this searching plays bad joke.
When CI picks up a new branch and notices, that a test had changed, it
spawns a custom job with test.py --repeat 100 $changed_test_name in
it. Next, when the test.py tries opt-in test name matching, it uses the
wildcard search and can pick up extra unwanted tests into the run.
To solve this, the case-selection syntax is extended. Now if the caller
specifies `suite/test::*` as test, the test file is selected by exact
name match, but the specific test-case is not selected, the `*` makes it
run all cases.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closesscylladb/scylladb#18704
Require users to specify whether we want shard for reads or for writes
by switching to appropriate non-deprecated variant.
For example, shard_of() can be replaced with shard_for_reads() or
shard_for_writes().
The next_shard/token_for_next_shard APIs have only for-reads variant,
and the act of switching will be a testimony to the fact that the code
is valid for intra-node migration.
Tablet sharder is adjusted to handle intra-migration where a tablet
can have two replicas on the same host. For reads, sharder uses the
read selector to resolve the conflict. For writes, the write selector
is used.
The old shard_of() API is kept to represent shard for reads, and new
method is introduced to query the shards for writing:
shard_for_writes(). All writers should be switched to that API, which
is not done in this patch yet.
The request handler on replica side acts as a second-level
coordinator, using sharder to determine routing to shards. A given
sharder has a scope of a single topology version, a single
effective_replication_map_ptr, which should be kept alive during
writes.
We need a separate transition kind for intra node migration so that we
don't have to recover this information from replica set in an
expensive way. This information is needed in the hot path - in
effective_replicaiton_map, to not return the pending tablet replica to
the coordinator. From its perspective, replica set is not
transitional.
The transition will also be used to alter the behavior of the
sharder. When not in intra-node migration, the sharder should
advertise the shard which is either in the previous or next replica
set. During intra-node migration, that's not possible as there may be
two such shards. So it will return the shard according to the current
read selector.
If allow_write_both_read_old tablet transition stage fails, move
to cleanup_target stage before reverting migration.
It's a preparation for further patches which deallocate storage
group of a tablet during cleanup.
This commit adds OS support in version 6.0.
In addition, it removes the information about version 5.2, as this version is no longer supported, according to our policy.
Closesscylladb/scylladb#18562
This PR:
- Removes the `.. only:: opensource` directive from Consistent Topology with Raft.
This feature is no longer an Open Source-only experimental feature.
- Removes redundant version-specific information.
- Moves the necessary version-specific information to a separate file.
This is a follow-up to 55b011902e.
Refs https://github.com/scylladb/scylladb/pull/18285/Closesscylladb/scylladb#18553
Someone thought that they actually represent real keys (the 'EXAMPLE' in their name was not enough).
Converted them to be as clear as can be, example data.
Signed-off-by: Yaniv Kaul <yaniv.kaul@scylladb.com>
Closesscylladb/scylladb#18565
This is the second half of the fix for issue #13968. The first half is already merged with PR #18346
Scylla issues warnings for partitions containing more rows than a configured threshold. The warning is issued by inserting a row into the `system.large_partitions` table. This row contains the information about the partition for which the warning is issued: keyspace, table, sstable, partition key and size, compaction time and the number of rows in the partition. A previous PR #18346 also added range tombstone count to this row.
This change adds a new counter for dead rows to the large_partitions table.
This change also adds cluster feature protection for writing into these new counters. This is needed in case a cluster is in the process of being upgraded to this new version, after which an upgraded node writes data with the new schema into `system.large_partitions`, and finally a node is then rolled back to an old version. This node will then revert the schema to the old version, but the written sstables will still contain data with the new counters, causing any readers of this table to throw errors when they encounter these cells.
This is an enhancement, and backporting is not needed.
Fixes#13968Closesscylladb/scylladb#18458
* github.com:scylladb/scylladb:
sstable: added test for counting dead rows
sstable: added docs for system.large_partitions.dead_rows
sstable: added cluster feature for dead rows and range tombstones
sstable: write dead_rows count to system.large_partitions
sstable: added counter for dead rows
This pull request introduces host ID in the Hinted Handoff module. Nodes are now identified by their host IDs instead of their IPs. The conversion occurs on the boundary between the module and `storage_proxy.hh`, but aside from that, IPs have been erased.
The changes take into considerations that there might still be old hints, still identified by IPs, on disk – at start-up, we map them to host IDs if it's possible so that they're not lost.
Refs scylladb/scylladb#6403Fixesscylladb/scylladb#12278Closesscylladb/scylladb#15567
* github.com:scylladb/scylladb:
docs: Update Hinted Handoff documentation
db/hints: Add endpoint_downtime_not_bigger_than()
db/hints: Migrate hinted handoff when cluster feature is enabled
db/hints: Handle arbitrary directories in resource manager
db/hints: Start using hint_directory_manager
db/hints: Enforce providing IP in get_ep_manager()
db/hints: Introduce hint_directory_manager
db/hints/resource_manager: Update function description
db/hints: Coroutinize space_watchdog::scan_one_ep_dir()
db/hints: Expose update lock of space watchdog
db/hints: Add function for migrating hint directories to host ID
db/hints: Take both IP and host ID when storing hints
db/hints: Prepare initializing endpoint managers for migrating from IP to host ID
db/hints: Migrate to locator::host_id
db/hints: Remove noexcept in do_send_one_mutation()
service: Add locator::host_id to on_leave_cluster
service: Fix indentation
db/hints: Fix indentation
Finalization of tablet split was only synchronizing with migrations, but
that's not enough as we want to make sure that all processes like repair
completes first as they might hold erm and therefore will be working
with a "stale" version of token metadata.
For synchronization to work properly, handling of tablet split finalize
will now take over the state machine, when possible, and execute a
global token metadata barrier to guarantee that update in topology by
split won't cause problems. Repair for example could be writing a
sstable with stale metadata, and therefore, could generate a sstable
that spans multiple tablets. We don't want that to happen, therefore
we need the barrier.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Closesscylladb/scylladb#18380
In order to correctly restore schema from `DESC SCHEMA WITH INTERNALS`, we need a way to drop a column with a timestamp in the past.
Example:
- table t(a int pk, b int)
- insert some data1
- drop column b
- add column b int
- insert some data2
If the sstables weren't compacted, after restoring the schema from description:
- we will loss column b in data2 if we simply do `ALTER TABLE t DROP b` and `ALTER TABLE t ADD b int`
- we will resurrect column b in data1 if we skip dropping and re-adding the column
Test for this: https://github.com/scylladb/scylla-dtest/pull/4122Fixes#16482Closesscylladb/scylladb#18115
* github.com:scylladb/scylladb:
docs/cql: update ALTER TABLE docs
test/cqlpytest: add test for prepared `ALTER TABLE ... DROP ... USING TIMESTAMP ?`
test/cql-pytest: remove `xfail` from alter table with timestamp tests
cql3/statements: extend `ALTER TABLE ... DROP` to allow specifying timestamp of column drop
cql3/statements: pass `query_options` to `prepare_schema_mutations()`
cql3/statements: add bound terms to alter table statement
cql3/statements: split alter_table_statement into raw and prepared
schema: allow to specify timestamp of dropped column
We move consistent cluster management out of experimental and
make it the default for new clusters in 6.0. In code, we make the
`consistent-topology-changes` flag unused and assumed to be true.
In 6.0, the topology upgrade procedure will be manual and
voluntary, so some clusters will still be using the gossip-based
topology even though they support the raft-based topology.
Therefore, we need to continue testing the gossip-based topology.
This is possible by using the `force-gossip-topology-changes` flag
introduced in scylladb/scylladb#18284.
Ref scylladb/scylladb#17802Closesscylladb/scylladb#18285
* github.com:scylladb/scylladb:
docs: raft.rst: update after removing consistent-topology-changes
treewide: fix indentation after the previous patch
db: config: make consistent-topology-changes unused
test: lib: single_node_cql_env: restart a node in noninitial run_in_thread calls
test: test_read_required_hosts: run with force-gossip-topology-changes
storage_service: join_cluster: replace force_gossip_based_join with force-gossip-topology-changes
storage_service: join_token_ring: fix finish_setup_after_join calls
dclocal_read_repair_chance and read_repair_chance have been removed in Cassandra 3.11 and 4.x, see
https://issues.apache.org/jira/browse/CASSANDRA-13910. if we expose these properties via DDL, Cassandra would fail to consume the CQL statement creating the table when performing migration from Scylla to Cassandra 4.x, as the latter does not understand these properties anymore.
currently the default values of `dc_local_read_repair_chance` and `read_repair_chance` are both "0". so they are practically disabled, unless user deliberately set them to a value greater than 0.
also, as a side effect, Cassandra 4.x has better support of Python3. the cqlsh shipped along with Cassandra 3.11.16 only supports python2.7, see
https://github.com/apache/cassandra/blob/cassandra-3.11.16/bin/cqlsh.py it errors out if the system only provides python3 with the error of
```
No appropriate python interpreter found.
```
but modern linux systems do not provide python2 anymore.
so, in this change, we deprecate these two options.
Fixes#3502
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#18087
* github.com:scylladb/scylladb:
docs: drop documents related to {,dclocal_}read_repair_chance
treewide: remove {dclocal_,}read_repair_chance options
