16ee3b3ebe
"
This series attempts to solve the regressions recently discovered in
performance of multi-partition range-scans. Namely that they:
* Flood the reader concurrency semaphore's queues, trampling other
reads.
* Behave very badly when too many of them is running concurrently
(trashing).
* May deadlock if enough of them is running without a timeout.
The solution for these problems is to make inactive shard readers
evictable. This should address all three issues listed above, to varying
degrees:
* Shard readers will now not cling onto their permits for the entire
duration of the scan, which might be a lot of time.
* Will be less affected by infinite concurrency (more than the node can
handle) as each scan now can make progress by evicting inactive shard
readers belonging to other scans.
* Will not deadlock at all.
In addition to the above fix, this series also bundles two further
improvements:
* Add a mechanism to `reader_concurrecy_semaphore` to be notified of
newly inserted evictables.
* General cleanups and fixes for `multishard_combining_reader` and
`foreign_reader`.
I can unbundle these mini series and send them separately, if the
maintainers so prefer, altough considering that this series will have to
be backported to 3.0, I think this present form is better.
Fixes: #3835
"
* 'evictable-inactive-shard-readers/v7' of https://github.com/denesb/scylla: (27 commits)
tests/multishard_mutation_query_test: test stateless query too
tests/querier_cache: fail resource-based eviction test gracefully
tests/querier_cache: simplify resource-based eviction test
tests/mutation_reader_test: add test_multishard_combining_reader_next_partition
tests/mutation_reader_test: restore indentation
tests/mutation_reader_test: enrich pause-related multishard reader test
multishard_combining_reader: use pause-resume API
query::partition_slice: add clear_ranges() method
position_in_partition: add region() accessor
foreign_reader: add pause-resume API
tests/mutation_reader_test: implement the pause-resume API
query_mutations_on_all_shards(): implement pause-resume API
make_multishard_streaming_reader(): implement the pause-resume API
database: add accessors for user and streaming concurrency semaphores
reader_lifecycle_policy: extend with a pause-resume API
query_mutations_on_all_shards(): restore indentation
query_mutations_on_all_shards(): simplify the state-machine
multishard_combining_reader: use the reader lifecycle policy
multishard_combining_reader: add reader lifecycle policy
multishard_combining_reader: drop unnecessary `reader_promise` member
...
(cherry picked from commit 414b14a6bd)
67 lines
2.5 KiB
C++
67 lines
2.5 KiB
C++
/*
|
|
* Copyright (C) 2015 ScyllaDB
|
|
*/
|
|
|
|
/*
|
|
* This file is part of Scylla.
|
|
*
|
|
* Scylla is free software: you can redistribute it and/or modify
|
|
* it under the terms of the GNU Affero General Public License as published by
|
|
* the Free Software Foundation, either version 3 of the License, or
|
|
* (at your option) any later version.
|
|
*
|
|
* Scylla is distributed in the hope that it will be useful,
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
* GNU General Public License for more details.
|
|
*
|
|
* You should have received a copy of the GNU General Public License
|
|
* along with Scylla. If not, see <http://www.gnu.org/licenses/>.
|
|
*/
|
|
|
|
#pragma once
|
|
|
|
#include "mutation_reader.hh"
|
|
#include "tests/sstable_utils.hh"
|
|
|
|
using populate_fn = std::function<mutation_source(schema_ptr s, const std::vector<mutation>&)>;
|
|
|
|
// Must be run in a seastar thread
|
|
// If fwd_sm == no, intra-partition (streamed_mutation) fast-forwarding tests
|
|
// will be skipped.
|
|
void run_mutation_source_tests(populate_fn populate, streamed_mutation::forwarding fwd_sm = streamed_mutation::forwarding::yes);
|
|
|
|
enum are_equal { no, yes };
|
|
|
|
// Calls the provided function on mutation pairs, equal and not equal. Is supposed
|
|
// to exercise all potential ways two mutations may differ.
|
|
void for_each_mutation_pair(std::function<void(const mutation&, const mutation&, are_equal)>);
|
|
|
|
// Calls the provided function on mutations. Is supposed to exercise as many differences as possible.
|
|
void for_each_mutation(std::function<void(const mutation&)>);
|
|
|
|
// Returns true if mutations in schema s1 can be upgraded to s2.
|
|
inline bool can_upgrade_schema(schema_ptr from, schema_ptr to) {
|
|
return from->is_counter() == to->is_counter();
|
|
}
|
|
|
|
class random_mutation_generator {
|
|
class impl;
|
|
std::unique_ptr<impl> _impl;
|
|
public:
|
|
struct generate_counters_tag { };
|
|
using generate_counters = bool_class<generate_counters_tag>;
|
|
|
|
explicit random_mutation_generator(generate_counters, local_shard_only lso = local_shard_only::yes);
|
|
~random_mutation_generator();
|
|
mutation operator()();
|
|
// Generates n mutations sharing the same schema nad sorted by their decorated keys.
|
|
std::vector<mutation> operator()(size_t n);
|
|
schema_ptr schema() const;
|
|
clustering_key make_random_key();
|
|
std::vector<dht::decorated_key> make_partition_keys(size_t n);
|
|
std::vector<query::clustering_range> make_random_ranges(unsigned n_ranges);
|
|
};
|
|
|
|
bytes make_blob(size_t blob_size);
|