Queries with query page size equal or smaller than
zero are unpaged queries.
Count these kind of queries and make them a metrics
since they can ruin the performance of the system.
Message-Id: <20170731130004.25807-2-benoit@scylladb.com>
"This series reduce that effect in two ways:
1. Remove the latency counters from the system keyspaces
2. Reduce the histogram size by limiting the maximum number of buckets and
stop the last bucket."
Fixes#2650.
* 'amnon/remove_cf_latency_v2' of github.com:cloudius-systems/seastar-dev:
database: remove latency from the system table
estimated histogram: return a smaller histogram
Since the discovery of std::exchange(x, {}) move_and_clear has become
obsolete. Beside, the name was wrong, it did not clear the vector but
recreated it meaning that any allocated memory wasn't reused (not that
it mattered in the existing usages).
Message-Id: <20170731123549.10887-1-pdziepak@scylladb.com>
"These patches optimise combined_mutation_reader for cases where the majority
of mutation_readers is disjoint.
perf_fast_forward:
Results are medians of 3 of fragments/s as reported by perf_fast_forward.
Command:
perf_fast_forward -c1 --enable-cache
small: small-partition-skips (read=1, skip=0)
large: large-partition-skips (read=1, skip=0)
before after diff
small 195753 238196 +22%
large 1244325 1359096 +9%
perf_simple_query:
Results are medians of 10 of reads/s as reported by perf_simple_query.
Command:
perf_simple_query -c1
before 98651.40
after 104554.85
diff +6%"
* tag 'avoid-merge_mutations/v1' of https://github.com/pdziepak/scylla:
combined_mutation_reader: avoid unnecessary merge_mutations()
combined_mutation_reader: do not pop mutation with different key
"This series ensure that when we retry a memtable flush, we re-acquire the
flush permit that was previously released. It also ensures we don't hold
the sstable read lock for the duration of the sleep leading to the retry.
To achieve that cleanly we refactor the way the permit lifecycle is managed
by employing a RAII-based approach.
We also improve the latency of writes blocked on virtual dirty by releasing
the flush permit before fsyncing the sstables. There are additional avenues
for performance improvements on top of this one."
* 'memtable-flush-additional-fixes/v4' of github.com:duarten/scylla:
column_family: Re-acquire flush permit in case of error
column_family: Don't hold sstable read lock when retrying flush
sstables: Release the flush permit before fsyncing
sstables: Introduce write_monitor
database: Extract out dirty_memory_manager
dirty_memory_manager: Refactor flush permit lifetime management
dirty_memory_manager: Invert permit acquisition order
memtable_list: Register different seal functions for each behaviour
Merging mutations is quite an expensive operation. The creation of
streamed mutation merger involves several allocations (mostly coming
from various std::vector) and then all mutation_fragments need to go
through a heap.
All this is completely unnecessary if there is only one mutation, so
let's skip a call to merge_mutations() in such cases. This also means
that we can reuse memory allocated by _current vector if merge is not
required.
Originally, the loop insidecombined_mutation_reader::next() so that it
was popping mutation from the heap and when it encountered one with a
different decorated key it was pushed back and the ones accumulated so
far merged and emitted. In other words, every time the reader progressed
to the next mutation it did needless pop and push operations on the
heap.
This patch rearranges the code so that the key of the next mutation is
compared before it is popped from the heap.
If we fail to flush an sstable, after creating the flush_reader, then
we will have released the flush permit when we retry the flush. Ensure
that when retrying, we re-acquire the flush permit.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This allows a queued flush to start while we fsync the current
sstable, which helps reduce the overall time new writes are blocked on
dirty memory.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
The write_monitor provides callbacks to inform an observer of the
state of the ongoing sstable write.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch refactors how the flush permit lifetime is managed,
dropping the current hash table in favour of a RAII approach.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
For an upcoming fix it is required to invert the permit acquisition
order: first we acquire the background work permit and then the single
flush permit.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Instead of passing a flush_behaviour to the seal function, use two
different functions for each of the behaviours.
This will be important in the forthcoming patches, which will require
the signatures of those functions to differ.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
serialized_action_tests depends on the fact that first part of the
serialized_action is executed at cetrtain points (in which it reads a
global variable that is later updated by the main thread).
This worked well in the release mode before ready continuations were
inlined and run immediately, but not in the debug mode since inlining
was not happening and the main seastar::thread was missing some yield
points.
Message-Id: <20170731103013.26542-1-pdziepak@scylladb.com>
1. assert() is not constexpr.
2. can't use static_assert(), because the contructor may be called in a non-constexpr
environment; moved to log_histogram
3. pow2_rank() uses count_leading_zeros() which is not constexpr; split
into constexpr and non-constexpr versions
4. duplicated number_of_buckets() because bucket_of() can't be constexpr due to pow2_rank
Message-Id: <20170726105444.32698-1-avi@scylladb.com>
"This series ensure that when we retry a memtable flush, we re-acquire the
flush permit that was previously released. It also ensures we don't hold
the sstable read lock for the duration of the sleep leading to the retry.
To achieve that cleanly we refactor the way the permit lifecycle is managed
by employing a RAII-based approach.
We also improve the latency of writes blocked on virtual dirty by releasing
the flush permit before fsyncing the sstables. There are additional avenues
for performance improvements on top of this one."
* 'memtable-flush-additional-fixes/v3' of github.com:duarten/scylla:
column_family: Re-acquire flush permit in case of error
column_family: Don't hold sstable read lock when retrying flush
sstables: Release the flush permit before fsyncing
sstables: Introduce write_monitor
database: Extract out dirty_memory_manager
dirty_memory_manager: Refactor flush permit lifetime management
dirty_memory_manager: Invert permit acquisition order
memtable_list: Register different seal functions for each behaviour
main: Don't catch polymorphic exceptions by value
* seastar a14d667...54e940f (8):
> Merge "Prometheus to use output stream" from Amnon
> http_test: Fix an http output stream test
> build: harden try_compile_and_link output temporary file
> configure: disable exception scalability hack on debug build
> build: don't perform test compiles to /dev/null
> Provide workaround for non scaleable c++ exception runtime
> Merge "Add output stream to http message reply" from Amnon
> configure.py: use user provided compiler flags when checking for features
Commitlog replay fails when upgrade from <1.7.3 to 2.0, we need to refuse
updating package if current scylla < 1.7.3 && commitlog remains.
Note: We have the problem on scylla-server package, but to prevent
scylla-conf package upgrade, %pretrans should be define on scylla-conf.
Fixes#2551
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1501187555-4629-1-git-send-email-syuu@scylladb.com>
"This patch series backports scalability fix for _Unwind_Find_FDE and modifies
out CentOS package to use our libgcc and libstdc++ which are needed to make
use of the fix instead of locally installed ones."
Ref #2646 (fixes on RHEL 7 and related only)
* 'gleb/exception-gcc-fix-v2' of github.com:cloudius-systems/seastar-dev:
dist/redhat: Make scylla rpm depend on scylla-libgcc and scylla-libstdc++ and use it instead of locally installed one
dist/redhat: Backport scalability fix of _Unwind_Find_FDE to out gcc
If we fail to flush an sstable, after creating the flush_reader, then
we will have released the flush permit when we retry the flush. Ensure
that when retrying, we re-acquire the flush permit.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This allows a queued flush to start while we fsync the current
sstable, which helps reduce the overall time new writes are blocked on
dirty memory.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
The write_monitor provides callbacks to inform an observer of the
state of the ongoing sstable write.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch refactors how the flush permit lifetime is managed,
dropping the current hash table in favour of a RAII approach.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
For an upcoming fix it is required to invert the permit acquisition
order: first we acquire the background work permit and then the single
flush permit.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Instead of passing a flush_behaviour to the seal function, use two
different functions for each of the behaviours.
This will be important in the forthcoming patches, which will require
the signatures of those functions to differ.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
If schema merging completes at lower rate than incoming pull requests,
then merge processes will accumulate and needlessly request and hold schema mutations.
In rare cases, when there are constant schema changes, they may even
overflow memory. This was seen in dtest:
concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.create_lots_of_schema_churn_test
Allowing only one active and one queued pull request per remote
endpoint is enough.
* tag 'tgrabiec/dont-accumulate-schema-pulls-v2' of github.com:scylladb/seastar-dev:
migration_manager: Log schema pulls
migration_manager: Prevent pull requests from accumulating
utils: Introduce serialized_action
If schema merging completes at lower rate than incoming pull requests,
then merge processes will accumulate and needlessly request and hold schema mutations.
In rare cases, when there are constant schema changes, they may even
overflow memory. This was seen in dtest:
concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.create_lots_of_schema_churn_test
Allowing only one active and one queued pull request per remote
endpoint is enough.
consume_mutation_fragments_until() allows consuming mutation fragments
until a specified condition happens. This patch reorganises its
implementation so that we avoid situations when fill_buffer() is called
with stop condition being true.
Message-Id: <20170727122218.7703-1-pdziepak@scylladb.com>
