Describe the high-level scheme of managing RPC mappings and
also expand on the introduction of "expirable" RPC mappings concept
and why these are needed.
Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
Raft instance needs to update RPC subsystem on changes in
configuration, so that RPC can deliver messages to the new nodes
in configuration, as well as dispose of the old nodes.
I.e. the nodes which are not the part of the most recent
configuration anymore.
The effective scope of RPC mappings is limited by the piece of
code which sends messages to both the "new" nodes (which
are added to the cluster with the most recent configuration
change) and the "old" nodes which are removed from the cluster.
Until the messages are successfully delivered to at least
the majority of "old" nodes and we have heard back from them,
the mappings should be kept intact. After that point the RPC
mappings for the removed nodes are no longer of interest
and thus can be immediately disposed.
Tests: unit(dev)
Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
The field is set in `fsm.get_output` whenever
`_log.last_conf_idx()` or the term changes.
Also, add `_last_conf_idx` and `_last_term` to
`fsm::last_observed_state`, they are utilized
in the condition to evaluate current rpc
configuration in `fsm.get_output()`.
This will be used later to update rpc config state
stored in `server_impl` and maintain rpc address map.
Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
Introduce rpc server_address that represents the
last observed state of address mappings
for RPC module.
It does not correspond to any kind of configuration
in the raft sense, just an artificial construct
corresponding to the largest set of server
addresses coming from both previous and current
raft configurations (to be able to contact both
joining and leaving servers).
This will be used later to update rpc module mappings
when cluster configuration changes.
Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
`std::unordered_set::contains` is introduced in C++20 and provides
clearer semantics to check existence of a given element in a set.
Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
This patch introduces `raft_address_map` class to abstract
the notion of expirable address mappings for a raft rpc module.
In Raft an instance may need to communicate with a peer outside
its current configuration. This may happen, e.g., when a follower
falls out of sync with the majority and then a configuration is
changed and a leader not present in the old configuration is elected.
The solution is to introduce the concept of "expirable" updates to
the RPC subsystem.
When RPC receives a message from an unknown peer, it also adds the
return address of the peer to the address map with a TTL. Should
we need to respond to the peer, its address will be known.
An outgoing communication to an unconfigured peer is impossible.
Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
Seven etcd unit tests as boost tests.
* alejo/raft-tests-etcd-08-v4-communicate-v5:
raft: etcd unit tests: test proposal handling scenarios
raft: etcd unit tests: test old messages ignored
raft: etcd unit tests: test single node precandidate
raft: etcd unit tests: test dueling precandidates
raft: etcd unit tests: test dueling candidates
raft: etcd unit tests: test cannot commit without new term
raft: etcd unit tests: test single node commit
raft: etcd unit tests: update test_leader_election_overwrite_newer_logs
raft: etcd unit tests: fix test_progress_leader
raft: testing: log comparison helper functions
raft: testing: helper to make fsm candidate
raft: testing: expose log for test verification
raft: testing: use server_address_set
raft: testing: add prevote configuration
raft: testing: make become_follower() available for tests
TestProposal
For multiple scenarios, check proposal handling.
Note, instead of expecting an explicit result for each specified case,
the test automatically checks for expected behavior when quorum is
reached or not.
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
TestSingleNodePreCandidate
Checks a single node configuration with precandidate on works to
automatically elect the node.
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
TestDuelingPreCandidates
In a configuration of 3 nodes, two nodes don't see each other and they
compete for leadership. Loser (3) should revert to follower when prevote
is rejected and revert to term 1.
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
TestDuelingCandidates
In a configuration of 3 nodes, two nodes don't see each other and they
compete for leadership. Once reconnected, loser should not disrupt.
But note it will remain candidate with current algorithm without
prevoting and other fsms will not bump term.
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
TestCannotCommitWithoutNewTermEntry tests the entries cannot be
committed when leader changes, no new proposal comes in and ChangeTerm
proposal is filtered.
NOTE: this doesn't check committed but it's implicit for next round;
this could also use communicate() providing committed output map
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
Port etcd TestSingleNodeCommit
In a single node configuration elect the node, add 2 entries and check
number of committed entries.
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
Make test_leader_election_overwrite_newer_logs use newer communicate()
and other new helpers.
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
Make implementation follow closer to original test.
Use newer boost test helpers.
NOTE: in etcd it seems a leader's self progress is in PIPELINE state.
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
Two helper functions to compare logs. For now only index, term, and data
type are used. Data content comparison does not seem to be necessary for now.
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
Current election_timeout() helper might bump the term twice.
It's convenient and less error prone to have a more fine grained helper
that stops right when candidate state is reached.
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
One INSERT statement was unnecessary for the test, so delete it.
Another was necessary, so explain it.
Tests: cql-pytest/test_null on both Scylla and Cassandra
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Closes#8304
Test raft configuration changes:
a node with empty configuration, transitioning
to an entirely different cluster, transitioning
in presence of down nodes, leader change during
configuration change, stray replies, etc.
* scylla-dev/raft-empty-confchange-v5: (21 commits)
raft: (testing) stray replies from removed followers
raft: always return a non-zero configuration index from the log
raft: (testing) leader change during configuration change
raft: (testing) test confchange {ABCDE} -> {ABCDEFG}
raft: (testing) test confchange {ABCDEF} -> {ABCGH}
raft: (testing) test confchange {ABC} -> {CDE}
raft: (testing) test confchange {AB} -> {CD}
raft: (testing) test confchange {A} -> {B}
raft: (testing) test a server with empty configuration
raft: (testing) introduce testing utilities
raft: (testing) simplify id allocation in test
raft: (testing) add select_leader() helper
raft: (testing) introduce communicate() helper
raft: (testing) style cleanup in raft_fsm_test
raft: (testing) fix bug in election_threshold
raft: minor style changes & comments
raft: do not assert when transitioning to empty config
raft: assert we never apply a snapshot over uncommitted entries (leader)
raft: improve tracing
raft: add fsm_output::empty() helper to aid testing
...
Right now toppartitions can only be invoked on one column family at a time.
This change introduces a natural extension to this functionality,
allowing to specify a list of families.
We provide three ways for filtering in the query parameter "name_list":
1. A specific column family to include in the form "ks:cf"
2. A keyspace, telling the server to include all column families in it.
Specified by omitting the cf name, i.e. "ks:"
3. All column families, which is represented by an empty list
The list can include any amount of one or both of the 1. and 2. option.
Fixes#4520Closes#7864
LCS reshape is basically 'major compacting' level 0 until it contains less than
N sstables.
That produces terrible write amplification, because any given byte will be
compacted (initial # of sstables / max_threshold (32)) times. So if L0 initially
contained 256 ssts, there would be a WA of about 8.
This terrible write amplification can be reduced by performing STCS instead on
L0, which will leave L0 in a good shape without hurting WA as it happens
now.
Fixes#8345.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20210322150655.27011-1-raphaelsc@scylladb.com>
During the retry mechanism, it's possible to encounter a gate
closed exception, which should simply be ignored, because
it indicates that the server is shutting down.
Closes#8337
This open option tells seastar that the file in question
will be truncated to the needed size right at once and all
the subsequent writes will happen within this size. This
hint turns off append optimization in seastar that's not
that cheap and helps so save few cpu cycles.
The option was introduced in seastar by 8bec57bc.
tests: unit(dev), dtest(commitlog:
test_batch_commitlog,
test_periodic_commitlog,
test_commitlog_replay_on_startup)
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Message-Id: <20210323115409.31215-1-xemul@scylladb.com>
Add comments explaining the rationale from transfer_leadership()
(more PhD quotes), encapsulate stable leader check in tick()
into a lambda and add more detailed comments to it.
Until clang figures things out with the now infamous
`-llvm -inline-threshold X` parameter, let's allow customizing
it to make the compilation of release builds less tiresome.
For instance, scylla's row_level.o object file currently does not compile
for me until I decrease the inline threshold to a low value (e.g. 50).
Message-Id: <54113db9438e3c3371410996f49b7fbe9a1b7257.1616422536.git.sarna@scylladb.com>
Row marker has a cell name which sorts after the row tombstone's start
bound. The old code was writing the marker first, then the row
tombstone, which is incorrect.
This was harmeless to our sstable reader, which recognized both as
belonging to the current clustering row fragment, and collects both
fine.
However, if both atoms trigger creation of promoted index blocks, the
writer will create a promoted index with entries wich violate the cell
name ordering. It's very unlikely to run into in practice, since to
trigger promoted index entries for both atoms, the clustering key
would be so large so that the size of the marker cell exceeds the
desired promoted index block size, which is 64KB by default (but
user-controlled via column_index_size_in_kb option). 64KB is also the
limit on clustering key size accepted by the system.
This was caught by one of our unit tests:
sstable_conforms_to_mutation_source_test
...which runs a battery of mutation reader tests with various
desired promoted index block sizes, including the target size of 1
byte, which triggers an entry for every atom.
The test started to fail for some random seeds after commit ecb6abe
inside the
test_streamed_mutation_forwarding_is_consistent_with_slicing test
case, reporting a mutation mismatch in the following line:
assert_that(*sliced_m).is_equal_to(*fwd_m, slice_with_ranges.row_ranges(*m.schema(), m.key()));
It compares mutations read from the same sstable using different
methods, slicing using clustering key restricitons, and fast
forwarding. The reported mismatch was that fwd_m contained the row
marker, but sliced_m did not. The sstable does contain the marker, so
both reads should return it.
After reverting the commit which introduced dynamic adjustments, the
test passes, but both mutations are missing the marker, both are
wrong!
They are wrong because the promoted index contians entries whose
starting positions violate the ordering, so binary search gets confused
and selects the row tombstone's position, which is emitted after the
marker, thus skipping over the row marker.
The explanation for why the test started to fail after dynamic
adjustements is the following. The promoted index cursor works by
incrementally parsing buffers fed by the file input stream. It first
parses the whole block and then does a binary search within the parsed
array. The entries which cursor touches during binary search depend on
the size of the block read from the file. The commit which enabled
dynamic adjustements causes the block size to be different for
subsequent reads, which allows one of the reads to walk over the
corrupted entries and read the correct data by selecting the entry
corresponding to the row marker.
Fixes#8324
Message-Id: <20210322235812.1042137-1-tgrabiec@scylladb.com>
Clustering rows are now stored in intrusive btree, cells are
now stored in radix tree, but scylla-gdb tries to walk the
intrusive_set and vector/set union respectively.
For the former case -- the btree wrapper is introduced.
For the latter -- compiler optimizes-away too many important
bits and walking the tree turns into a bunch of hard-coded
hacks and reiterpret-casts. Untill better solution is found,
just print the address of the tree root.
* xemul/br-gdb-btree-rows:
gdb: Show address of the row::_cells tree (or "empty" mark)
gdb: Add support for intrusive B tree
gdb: Use helper to get rows from mutation_partition
Currently clang optimizes-out lots of critical stuff from
compact radix tree. Untill we find out the way to walk the
tree in gdb, it's better to at least show where it is in
memory.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
