db/hints: Introduce an error injection to test draining
We want to verify that a hint directory is drained when any of the nodes correspodning to it leaves the cluster. The test scenario should happen before the whole cluster has been migrated to the host-ID-based hinted handoff, so when we still rely on the mappings between hint endpoint managers and the hint directories managed by them. To make such a test possible, in these changes we introduce an error injection rejecting incoming hints. We want to test a scenario when: 1. hints are saved towards a given node -- node N1, 2. N1 changes its IP to a different one, 3. some other node -- node N2 -- changes its IP to the original IP of N1, 4. hints are saved towards N2 and they are stored in the same directory as the hints saved towards N1 before, 5. we start draining N2. Because at some point N2 needs to be stopped, it may happen that some mutations towards a distributed system table generate a hint to N2 BEFORE it has finished changing its IP, effectively creating another hint directory where ALL of the hints towards the node will be stored from there on. That would disturb the test scenario. Hence, this error injection is necessary to ensure that all of the steps in the test proceed as expected.
This commit is contained in:
@@ -409,6 +409,12 @@ bool manager::have_ep_manager(const std::variant<locator::host_id, gms::inet_add
|
||||
bool manager::store_hint(endpoint_id host_id, gms::inet_address ip, schema_ptr s, lw_shared_ptr<const frozen_mutation> fm,
|
||||
tracing::trace_state_ptr tr_state) noexcept
|
||||
{
|
||||
if (utils::get_local_injector().enter("reject_incoming_hints")) {
|
||||
manager_logger.debug("Rejecting a hint to {} / {} due to an error injection", host_id, ip);
|
||||
++_stats.dropped;
|
||||
return false;
|
||||
}
|
||||
|
||||
if (stopping() || draining_all() || !started() || !can_hint_for(host_id)) {
|
||||
manager_logger.trace("Can't store a hint to {}", host_id);
|
||||
++_stats.dropped;
|
||||
|
||||
Reference in New Issue
Block a user