Improves memtable flush performance by 10% in a CPU-bound case.
Unlike the zlib implementation, libdeflate is optimized for modern
CPUs. It utilizes the PCLMUL instruction.
checksum_combine() is much slower than re-feeding the buffer to
checksum() for the zlib CRC32 checksummer.
Introduce Checksum::prefer_combine() to determine this and select
more optimal behavior for given checksummer.
Improves performance of memtable flush with compression enabled by 30%.
"
Some queries are very unlikely to hit cache. Usually this includes
range queries on large tables, but other patterns are possible.
While the database should adapt to the query pattern, sometimes the
user has information the database does not have. By passing this
information along, the user helps the database manage its resources
more optimally.
To do this, this patch introduces a BYPASS CACHE clause to the
SELECT statement. A query thus marked will not attempt to read
from the cache, and instead will read from sstables and memtables
only. This reduces CPU time spent to query and populate the cache,
and will prevent the cache from being flooded with data that is
not likely to be read again soon. The existing cache disabled path
is engaged when the option is selected.
Tests: unit (release), manual metrics verification with ccm with and without the
BYPASS CACHE clause.
Ref #3770.
"
* tag 'cache-bypass/v2' of https://github.com/avikivity/scylla:
doc: document SELECT ... BYPASS CACHE
tests: add test for SELECT ... BYPASS CACHE
cql: add SELECT ... BYPASS CACHE clause
db: add query option to bypass cache
* tag 'perf-ffwd-dataset-population-v2' of github.com:tgrabiec/scylla:
tests: perf_fast_forward: Measure performance of dataset population
tests: perf_fast_forward: Record the dataset on which test case was run
tests: perf_fast_forward: Introduce the concept of a dataset
tests: perf_fast_forward: Introduce make_compaction_disabling_guard()
tests: perf_fast_forward: Initialize output manager before population
tests: perf_fast_forward: Handle empty test parameter set
tests: perf_fast_forward: Extract json_output_writer::write_common_test_group()
tests: perf_fast_forward: Factor out access to cfg to a single place per function
tests: perf_fast_forward: Extract result_collector
tests: perf_fast_forward: Take writes into account in AIO statistics
tests: perf_fast_forward: Reorder members
tests: perf_fast_forward: Add --sstable-format command line option
The BYPASS CACHE clause instructs the database not to read from or populate the
cache for this query. The new keywords (BYPASS and CACHE) are not reserved.
do_process_buffer had two unreachable default cases and a long
if-else-if chain.
This converts the the if-else-if chain to a switch and a helper
function.
This moves the error checking from run time to compile time. If we
were to add a 128 bit integer for example, gcc would complain about it
missing from the switch.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20181125221451.106067-1-espindola@scylladb.com>
"
This new compaction approach consists of releasing exhausted fragments[1] of a run[2] a
compaction proceeds, so decreasing considerably the space requirement.
These changes will immediately benefit leveled strategy because it already works with
the run concept.
[1] fragment is a sstable composing a run; exhausted means sstable was fully consumed
by compaction procedure.
[2] run is a set of non-overlapping sstables which roughly span the
entire token range.
Note:
Last patch includes an example compaction strategy showing how to work with the interface.
unit tests: all modes passing
dtests: compaction ones passing
"
* 'sstable_run_based_compaction_v10' of github.com:raphaelsc/scylla:
tests: add example compaction strategy for sstable run based approach
sstables/compaction: propagate sstable replacement to all compaction of a CF
sstables: store cf pointer in compaction_info
tests/sstable_test: add test for compaction replacement of exhausted sstable
sstables: add sstable's on closed handling
tests/sstables: add test for sstable run based compaction
sstables/compaction_manager: prevent partial run from being selected for compaction
compaction: use same run identifier for sstables generated by same compaction
sstables: introduce sstable run
sstables/compaction_manager: release reference to exhausted sstable through callback
sstables/compaction: stop tracking exhausted input sstable in compaction_read_monitor
database: do not keep reference to sstable in selector when done selecting
compaction: share sstable set with incremental reader selector
sstables/compaction: release space earlier of exhausted input sstables
sstables: make partitioned sstable set's incremental selector resilient to changes in the set
database: do not store reference to sstable in incremental selector
tests/sstables: add run identifier correctness test
sstables: use a random uuid for sstables without run identifier
sstables: add run identifier to scylla metadata
This is needed for parallel compaction to work with sstable run based approach.
That's because regular compaction clones a set containing all sstables of its
column family. So compaction A can potentially hold a reference to a compacting
sstable of compaction B, so preventing compacting B from releasing its exhausted
sstable.
So all replacements are propagated to all compactions of a given column family,
and compactions in turn, including the one which initiated the propagation,
will do the replacement.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
motivation is that we need a more efficient way to find compactions
that belong to a given column family in compaction list.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Make sure that compaction is capable of releasing exhausted sstable space
early in the procedure.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Motivation is that it will be useful for catching regression on compaction
when releasing early exhausted sstables. That's because sstable's space
is only released once it's closed. So this will allow us to write a test
case and possibly use it for entities holding exhausted sstable.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Filter out sstable belonging to a partial run being generated by an ongoing
compaction. Otherwise, that could lead to wrong decisions by the compaction
strategy.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
SSTables composing the same run will share the same run identifier.
Therefore, a new compaction strategy will be able to get all sstables belong
to the same run from sstable_set, which now keeps track of existing runs.
Same UUID is passed to writers of a given compaction. Otherwise, a new UUID
is picked for every sstable created by compaction.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
sstable run is a structure that will hold all sstables that has the same
run identifier. All sstables belonging to the same run will not overlap
with one another.
It can be used by compaction strategy to work on runs instead of individual
sstables.
sstable_set structure which holds all sstables for a given column family
will be responsible for providing to its user an interface to work with
runs instead of individual sstables.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
That's important for the reference to sstable to not be kept throughout
the compaction procedure, which would break the goal of releasing
space during compaction.
Manager passes a callback to compaction which calls it whenever
there's sstable replacement.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Motivation is that we want to release space for exhausted sstable and that
will only happen when all references to it are gone *and* that backlog
tracker takes the early replacement into account.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
When compacting, we'll create all readers at once and will not select
again from incremental selector, meaning the selector will keep all
respective sstables in current_sstables, preventing compaction from
releasing space as it goes on.
The change is about refreshing sstable set's selector such that it
will not hold a reference to an exhausted sstable whatsoever.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
By doing that, we'll be able to release exhausted sstable from both
simulteaneously.
That's achieved by sharing set containing input sstables with the incremental
reader selector and removing exhausted sstables from shared set when the
time has come.
Step towards reducing disk requirement for compaction by making it delete
sstable which all data is in a sealed new sstable. For that to happen,
all references must be gone.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Currently, compaction only replace input sstables at end of compaction,
meaning compaction must be finished for all the space of those sstables
to be released.
What we can do instead is to delete earlier some input sstable under
some conditions:
1) SStable data should be committed to a new, sealed output sstable,
meaning it's exhausted.
2) Exhausted sstable mustn't overlap with a non-exhausted sstable
because a tombstone in the exhausted could have been purged and the
shadowed data in non-exhausted could be ressurected if system
crashes.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
The motivation is that compaction may remove a sstable from the set while the
incremental selector is alive, and for that to work, we need to invalidate
the iterators stored by the selector. We could have added a method to notify
it, but there will be a case where the one keeping the set cannot forward
the notification to the selector. So it's better for the selector to take
care of itself. Change counter approach is used which allows the selector
to know when to invalidate the iterators.
After invalidation, selector will move the iterator back into its right
place by looking for lower bound for current pos.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Use sstable generation instead to keep track of read sstables.
The motivation is that we'll not keep reference to sstables, so allowing
their space on disk to be released as soon they get exhausted.
Generation is used because it guarantees uniqueness of the sstable.
Reviewed-by: Botond Dénes <bdenes@scylladb.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Older sstables must have an identifier for them to be associated
with their own run.
Reviewed-by: Nadav Har'El <nyh@scylladb.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
It identifies a run which a particular sstable belongs to.
Existing sstables will have a random uuid associated with it
in memory.
UUID is the correct choice because it allows sstables to be
exported without having conflicts when using identifier generated
by different nodes.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
These switches are fully covered. We can be sure they will stay this
way because of -Werror and gcc's -Wswitch warning.
We can also be sure that we never have an invalid enum value since the
state machine values are not read from disk.
The patch also removes a superfluous ';'.
Message-Id: <20181124020128.111083-1-espindola@scylladb.com>
The reason for that is that it's not available in sstable format mc,
so we can no longer rely on it in common code for the currently
supported formats.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20181121170057.20900-1-raphaelsc@scylladb.com>
A dataset represents a table with data, populated in certain way, with
certain characteristics of the schema and data.
Before this change, datasets were implicitly defined, with population
hard-coded inside the populate() function.
This change gathers logic related to datasets into classes, in order to:
- make it easier to define new datasets.
- be able to measure performance of dataset population in a
standardized way.
- being able to express constraints on datasets imposed by different
test cases. Test cases are matched with possible datasets based
on the abstract interface they accept (e.g. clustered_ds,
multipartition_ds), and which must be implemented by a compatible
dataset. To facilitate this matching, test function is now wrapped
into a dataset_acceptor object, with an automatically-generated can_run()
virtual method, deduced by make_test_fn().
- be able to select tests to run based on the dataset name.
Only tests which are compatible with that dataset will be run.
Extracts the result collection and reporting logic out of
run_test_case(). Will be needed in population tests, for which we
don't want the looping logic.
