We need to do that in order to close the database cleanly, flushing all pending
data before we do.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
This change abstracts reading from on-disk data sources behind a single
reader which is then composed with memtable readers. This change also
abstracts all data sources behind a single reader obtained via
column_family::make_reader(). That reader is then used by algorithms
like column_family::for_all_partitions() or
column_family::query(). Having those abstractions will make it easier
to add row cache, because it will be encapsulated in a single place.
partitions_ranges will be manipulated upon to be split for different
destination, so provide it separately from read_command to not copy the
later for each destination.
From Pekka:
"This series fixes up schema management code to store keyspace strategy
options to database. The map is stored as JSON just like in Origin."
Fix keyspace strategy options to preserve key-value ordering by
switching to std::map. We need this to be able to store the map in
database as JSON because unordered maps can cause the schema merging
code to attempt a keyspace update, which we don't support, even though
the values did not change.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Currently, we flush out memtables very aggressively which results into
lots of small sstable writes. The proper fix here is to do accounting on
the memtable size but before that happens, bump up the threshold to
another magic number which gives better batching:
$ ./build/release/seastar --smp 1 --data-file-directories data --commitlog-directory commitlog/
$ tools/bin/cassandra-stress write -mode cql3 native prepared -rate threads=32
Before:
Results:
op rate : 37280
partition rate : 37280
row rate : 37280
latency mean : 0.8
latency median : 0.6
latency 95th percentile : 1.1
latency 99th percentile : 7.6
latency 99.9th percentile : 11.9
latency max : 50.5
Total operation time : 00:00:30
END
After:
Results:
op rate : 46721
partition rate : 46721
row rate : 46721
latency mean : 0.7
latency median : 0.5
latency 95th percentile : 0.9
latency 99th percentile : 1.3
latency 99.9th percentile : 5.8
latency max : 96.3
Total operation time : 00:00:39
END
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Commit log guarantees that once an RP is assigned to a data frame/caller, it
will not block before returning the result via future. However, this is not
enough, since we could
a.) Have blocked earlier, in which case the return value processing will be
async anyway
b.) Even if no blocking takes place, future chaining mechanism could decide
it has to reorder execution.
Assuming though that the case where this happens is rare, and cases where it
actually affects the rule of replay position ordering is even rarer, we can
guard against it by simply keeping track of the highest RP _discarded_ (sent
to sstable flush), and if we attempt to apply a mutation with a higher RP,
simply re-do the operation (i.e. write same entry to commit log again).
Signed-off-by: Calle Wilund <calle@cloudius-systems.com>
- Forbid explicit snitch creation with constructor.
- Allow the creation of snitches only with locator::make_snitch() template
function.
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
New in v4:
- Make sure the snitch is stopped before it's destroyed when _snitch_is_ready
is returned in an exceptional state.
New in v2:
- Change snitch_ptr to be std::unique_ptr<i_endpoint_snitch>
- abstract_replication_strategy::create_replication_strategy(): explicitly
specify (template) types of create_object() parameters.
- Re-arrange the loop in marge_keyspaces() so that lambdas that depend on
"this" complete before there is a chance that "this" gets destroyed.
- create_keyspace(): Don't add a new keyspace if a keyspace with this name
already exists.
- i_endpoint_snitch: added a stop() virtual method
- Added a stop() pure virtual method.
- Added an enum class snitch_state and a _state member initialized to snitch_state::initializing,
added an assert() in a destructor requiring _state to become snitch_state::stopped,
which should be set when stop() is complete.
- rack_inferring_snitch: added a stop() method.
- simple_snitch: added a stop() method.
- Added stop() methods to abstract_replication_strategy and keyspace.
- Updated database::stop() to wait for all keyspaces in _keyspaces to stop.
* Forward commitlog replay_position to column_family.memtable, updating
highest RP if needed
* When flushing memtable, signal back to commitlog that RP has been dealt with
to potentially remove finished segment(s)
Note: since memtable flushing right now is _not_ explicitly ordered,
this does not actually work, since we need to guarantee ordering with
regards to RP. I.e. if we flush N blocks, we must guarantee that:
a.) We report "flushed RP" in RP order
b.) For a given RP1, all RP* lower than RP1 must also have been flushed.
(The latter means that it is fine to say, flush X tables at the same time, as long as we report a single RP that is the highest, and no lower RP:s exist in non-flushed tables)
I am however letting someone else deal with ensuring MT->sstable flush order.
Signed-off-by: Calle Wilund <calle@cloudius-systems.com>
Store a lw_shared_ptr<keyspace_metadata> in struct keyspace so callers
in migration manager, for example, can look it up.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
From Avi:
"This patchset prepares for adding sstables to the read path. Because sstables
involve I/O, their APIs return futures, which means that APIs that may call
those sstable APIs also need to return futures.
This patchset uses the two-space indent + do_with + reference aliases trick
to make patches more readable. Cleanup patches will follow once it is merged."
Initialize replication strategy when keyspace is being created now that
we have access to keyspace_metadata.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Use the keyspace_metadata type for keyspace creation functions. This is
needed to be able to have a mapping from keyspace name to keyspace
metadata for various call-sites.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Follow the naming convention set by user_types_metadata and rename
ks_meta_data to keyspace_metadata.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
The implementation part of ks_meta_data is just few lines of code.
Inline that to the database.hh header file.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
This is slightly awkwards, since the directory structure is not sharded.
This requires some processing to occur outside the shard, while the rest
is sharded.
Returning a reference to the keyspace is dangerous in that the keyspace can
be moved away, when we start futurizing the add_keyspace() process. Make
it return void and look up the keyspace at the point of use.
Following what happened to others: we can now include memtable.hh
without including database.hh
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
The immediate motivation for introducing frozen_mutation is inability
to deserialize current "mutation" object, which needs schema reference
at the time it's constructed. It needs schema to initialize its
internal maps with proper key comparators, which depend on schema.
frozen_mutation is an immutable, compact form of a mutation. It
doesn't use complex in-memory strucutres, data is stored in a linear
buffer. In case of frozen_mutation schema needs to be supplied only at
the time mutation partition is visited. Therefore it can be trivially
deserialized without schema.
