mirror of
https://github.com/scylladb/scylladb.git
synced 2026-04-26 11:30:36 +00:00
384a0628b0
Before this PR, writes to the previous CDC generations would
always be rejected. After this PR, they will be accepted if the
write's timestamp is greater than `now - generation_leeway`.
This change was proposed around 3 years ago. The motivation was
to improve user experience. If a client generates timestamps by
itself and its clock is desynchronized with the clock of the node
the client is connected to, there could be a period during
generation switching when writes fail. We didn't consider this
problem critical because the client could simply retry a failed
write with a higher timestamp. Eventually, it would succeed. This
approach is safe because these failed writes cannot have any side
effects. However, it can be inconvenient. Writing to previous
generations was proposed to improve it.
The idea was rejected 3 years ago. Recently, it turned out that
there is a case when the client cannot retry a write with the
increased timestamp. It happens when a table uses CDC and LWT,
which makes timestamps permanent. Once Paxos commits an entry
with a given timestamp, Scylla will keep trying to apply that entry
until it succeeds, with the same timestamp. Applying the entry
involves writing to the CDC log table. If it fails, we get stuck.
It's a major bug with an unknown perfect solution.
Allowing writes to previous generations for `generation_leeway` is
a probabilistic fix that should solve the problem in practice.
Apart from this change, this PR adds tests for it and updates
the documentation.
This PR is sufficient to enable writes to the previous generations
only in the gossiper-based topology. The Raft-based topology
needs some adjustments in loading and cleaning CDC generations.
These changes won't interfere with the changes introduced in this
PR, so they are left for a follow-up.
Fixes scylladb/scylladb#7251
Fixes scylladb/scylladb#15260
Closes scylladb/scylladb#17134
* github.com:scylladb/scylladb:
docs: using-scylla: cdc: remove info about failing writes to old generations
docs: dev: cdc: document writing to previous CDC generations
test: add test_writes_to_previous_cdc_generations
cdc: generation: allow increasing generation_leeway through error injection
cdc: metadata: allow sending writes to the previous generations
(cherry picked from commit 9bb4482ad0)
Backport note: in tests, replaced `servers_add` with loop of `server_add`
88 lines
3.7 KiB
C++
88 lines
3.7 KiB
C++
/*
|
|
* Copyright (C) 2019-present ScyllaDB
|
|
*/
|
|
|
|
/*
|
|
* SPDX-License-Identifier: AGPL-3.0-or-later
|
|
*/
|
|
|
|
#pragma once
|
|
|
|
#include <map>
|
|
|
|
#include "db_clock.hh"
|
|
#include "timestamp.hh"
|
|
#include "cdc/generation.hh"
|
|
|
|
namespace dht {
|
|
class token;
|
|
}
|
|
|
|
namespace cdc {
|
|
|
|
class stream_id;
|
|
class topology_description;
|
|
|
|
/* Represents the node's knowledge about CDC generations used in the cluster.
|
|
* Used during writes to pick streams to which CDC log writes should be sent to
|
|
* (i.e., to pick partition keys for these writes).
|
|
*/
|
|
class metadata final {
|
|
// Note: we use db_clock (1ms resolution) for generation timestaps
|
|
// (because we need to insert them into tables using columns of timestamp types,
|
|
// and the native type of our columns' timestamp_type is db_clock::time_point).
|
|
// On the other hand, timestamp_clock (1us resolution) is used for mutation timestamps,
|
|
// and api::timestamp_type represents the number of ticks of a timestamp_clock::time_point since epoch.
|
|
|
|
using container_t = std::map<api::timestamp_type, std::optional<topology_description>>;
|
|
container_t _gens;
|
|
|
|
/* The timestamp used in the last successful `get_stream` call. */
|
|
api::timestamp_type _last_stream_timestamp = api::missing_timestamp;
|
|
|
|
container_t::const_iterator gen_used_at(api::timestamp_type ts) const;
|
|
public:
|
|
/* Is a generation with the given timestamp already known or obsolete? It is obsolete if and only if
|
|
* it is older than the generation operating at `now - get_generation_leeway()`.
|
|
*/
|
|
bool known_or_obsolete(db_clock::time_point) const;
|
|
|
|
/* Are there streams available. I.e. valid for time == now. If this is false, any writes to
|
|
* CDC logs will fail fast.
|
|
*/
|
|
bool streams_available() const;
|
|
/* Return the stream for the base partition whose token is `tok` to which a corresponding log write should go
|
|
* according to the generation used at time `ts` (i.e, the latest generation whose timestamp is less or equal to `ts`).
|
|
*
|
|
* If the provided timestamp is too far away "into the future" (where "now" is defined according to our local clock),
|
|
* we reject the get_stream query. This is because the resulting stream might belong to a generation which we don't
|
|
* yet know about. Similarly, we reject queries to the previous generations if the timestamp is too far away "into
|
|
* the past". The amount of leeway (how much "into the future" or "into the past" we allow `ts` to be) is defined by
|
|
* `get_generation_leeway()`.
|
|
*/
|
|
stream_id get_stream(api::timestamp_type ts, dht::token tok);
|
|
|
|
/* Insert the generation given by `gen` with timestamp `ts` to be used by the `get_stream` function,
|
|
* if the generation is not already known or older than the currently known ones.
|
|
*
|
|
* Returns true if the generation was inserted,
|
|
* meaning that `get_stream` might return a stream from this generation (at some time points).
|
|
*/
|
|
bool insert(db_clock::time_point ts, topology_description&& gen);
|
|
|
|
/* Prepare for inserting a new generation whose timestamp is `ts`.
|
|
* This method is not required to be called before `insert`, but it's here
|
|
* to increase safety of `get_stream` calls in some situations. Use it if you:
|
|
* 1. know that there is a new generation, but
|
|
* 2. you didn't yet retrieve the generation's topology_description.
|
|
*
|
|
* After preparing a generation, if `get_stream` is supposed to return a stream from this generation
|
|
* but we don't yet have the generation's data, it will reject the query to maintain consistency of streams.
|
|
*
|
|
* Returns true iff this generation is not obsolete and wasn't previously prepared nor inserted.
|
|
*/
|
|
bool prepare(db_clock::time_point ts);
|
|
};
|
|
|
|
} // namespace cdc
|