mirror of
https://github.com/scylladb/scylladb.git
synced 2026-04-24 18:40:38 +00:00
df64985a4e
Current protocol extension that sends tablet info to drivers only does that if the driver selects a non-replica coordinator for a routable request. It works well if some node on the replica list is replaced by other node, or if some replicas are removed from the list. Driver will at some point send a request to stale replica, and receive new list in response. The issue is with extending the list with new replicas. In that case old replicas are all still correct, so driver will not select any wrong replica, and will not receive the new list. As far as I know that only scenario where this could happen is RF increase. It could be to some degree worked around in the drivers, but it would add significant complexity (definitely more than any other invalidations we introduced) while still not being ideal solution. This scenario should be rare enough, and the consequences of not handling it minor enough (new replicas not being used as coordinators) that it does not warrant driver-side solution. Instead this commit adds info about this to documentation, advising users to restart applications after replica lists are extended. It is worth noting that if new tablet feedback protocol extension is implemented then this problem goes away. See issue #21664. Closes scylladb/scylladb#23447
56 lines
2.6 KiB
ReStructuredText
56 lines
2.6 KiB
ReStructuredText
=======================================================
|
|
How to Safely Increase the Replication Factor
|
|
=======================================================
|
|
|
|
|
|
**Topic: What can happen when you increase RF**
|
|
|
|
|
|
**Audience: ScyllaDB administrators**
|
|
|
|
|
|
Issues
|
|
------
|
|
|
|
When a Replication Factor (RF) is increased, using the :ref:`ALTER KEYSPACE <alter-keyspace-statement>` command, the data consistency is effectively dropped
|
|
by the difference of the RF_new value and the RF_old value for all pre-existing data.
|
|
Consistency will only be restored after running a repair.
|
|
|
|
Another issue occurs in keyspaces with tablets enabled and is driver-related. Due to limitations in the current protocol used to pass tablet data to drivers, drivers will not pick
|
|
up new replicas after replication factor is increased. This will cause them to avoid routing requests to those replicas, causing imbalance.
|
|
|
|
Resolution
|
|
----------
|
|
|
|
When one increases an RF, one should consider that the pre-existing data will **not be streamed** to new replicas (a common misconception).
|
|
|
|
As a result, in order to make sure that you can keep on reading the old data with the same level of consistency, increase the read Consistency Level (CL) according to the following formula:
|
|
|
|
``CL_new = CL_old + RF_new - RF_old``
|
|
|
|
After you run a repair, you can decrease the CL. If RF has only been changed in a particular Data Center (DC) only the nodes in that DC have to be repaired.
|
|
|
|
To resolve the driver-related issue, restart the client applications after the ALTER statement that changes the RF completes successfully.
|
|
|
|
Example
|
|
=======
|
|
|
|
In this example your five node cluster RF is 3 and your CL is TWO. You want to increase your RF from 3 to 5.
|
|
|
|
#. Increase the read CL by a RF_new - RF_old value.
|
|
Following the example the RF_new is 5 and the RF_old is 3 so, 5-3 =2. You need to increase the CL by 2.
|
|
As the CL is currently TWO, increasing it more would be QUORUM, but QUORUM would not be high enough. For now, increase it to ALL.
|
|
#. Change the RF to RF_new. In this example, you would increase the RF to 5.
|
|
#. Repair all tables of the corresponding keyspace. If RF has only been changed in a particular Data Center (DC), only the DC nodes have to be repaired.
|
|
#. Restore the reads CL to the originally intended value. For this example, QUORUM.
|
|
|
|
|
|
If you do not follow the procedure above you may start reading stale or null data after increasing the RF.
|
|
|
|
More Information
|
|
----------------
|
|
|
|
* :doc:`Fault Tolerance </architecture/architecture-fault-tolerance/>`
|
|
* :ref:`Set the RF (first time) <create-keyspace-statement>`
|
|
* :ref:`Change the RF (increase or decrease) <alter-keyspace-statement>`
|