From Glauber:
"Shlomi found a bug when writing summary files under certain conditions.
The current patchset fixes it, while also performing some generally useful
cleanups."
From Pekka:
"This series improves the CQL test for 'create table' statement. It
hardens the test_create_statement() test case and converts
create_table() calls to use CQL where possible. Please note that we need
to go through the remaining create_table() call-sites and fix potential
issues in the CQL front-end or schema management code."
Add a require_table_exists() assertion after 'create table' statement is
executed to ensure in-memory schema is updated accordingly.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
We have recently had a bug where we were messing up with the summary generation
in certain corner cases. In particular, when the number of keys was an exact multiple
of the sampling rate.
Add a test case to make sure we never regress on this.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
"only lz4 is supported so far, but other algorithms should be fairly easy to
add, given that the infrastructure for compression is introduced here.
NOTE:
our sstable read code already reads compressed data file, so a test was added
to guarantee that data from a compressed sstable generated by our write code
can be successfully loaded."
To test the messaging service API it needs to be included in the
messaging service test.
To test it, start the server with --stay-alive true
then you can use the API to get the messges information:
http://localhost:10001/messaging_service/messages/exception
will return the number of exception per connection
http://localhost:10001/messaging_service/command/completed
will return the number of completed command per connection.
Note that because the two servers are running on the same machine, to
prevent port conflict, the server port will be increment by one (i.e
10001)
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
From Vlad:
"Currently database always created a SimpleSnitch and ignores the corresponding parameter
provided by the user. This series fixes this situation:
- Changes the snitch creation interface to comply the Java-like interface that
has already been used in a topology_strategy classes family.
- Fix all the places where a SimpleSnitch has been created ignoring the user configuration."
Fix keyspace strategy options to preserve key-value ordering by
switching to std::map. We need this to be able to store the map in
database as JSON because unordered maps can cause the schema merging
code to attempt a keyspace update, which we don't support, even though
the values did not change.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
- Forbid explicit snitch creation with constructor.
- Allow the creation of snitches only with locator::make_snitch() template
function.
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
New in v4:
- Make sure the snitch is stopped before it's destroyed when _snitch_is_ready
is returned in an exceptional state.
New in v2:
- Change snitch_ptr to be std::unique_ptr<i_endpoint_snitch>
- abstract_replication_strategy::create_replication_strategy(): explicitly
specify (template) types of create_object() parameters.
- Re-arrange the loop in marge_keyspaces() so that lambdas that depend on
"this" complete before there is a chance that "this" gets destroyed.
- create_keyspace(): Don't add a new keyspace if a keyspace with this name
already exists.
- i_endpoint_snitch: added a stop() virtual method
- Added a stop() pure virtual method.
- Added an enum class snitch_state and a _state member initialized to snitch_state::initializing,
added an assert() in a destructor requiring _state to become snitch_state::stopped,
which should be set when stop() is complete.
- rack_inferring_snitch: added a stop() method.
- simple_snitch: added a stop() method.
- Added stop() methods to abstract_replication_strategy and keyspace.
- Updated database::stop() to wait for all keyspaces in _keyspaces to stop.
The field compressor is about saying which compressor algorithm
must be used in compression of sstable data file.
This is a small step towards compressed sstable data file.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
For some reason, I added a fsync call when the file underlying the
stream gets truncated. That happens when flushing a file, which
size isn't aligned to the requested DMA buffer.
Instead, fsync should only be called when closing the stream, so this
patch changes the code to do that.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
When a promise that still tracks its future is destroyed, but after
future::get() or future::then() was called, the promise thinks it was
destroyed before generating a value, and fails an assertion.
Fix be detaching the promise from the future as soon as get() or then()
remove the value.
cql_query_test segfaults in debug mode due to infinite recursion (see
trace below). The problem started to appear after Avi's always-defer
change.
It looks like the following boost::any() constructor template is
instantiated when boost::any is copied into a lambda:
// Perfect forwarding of ValueType
template<typename ValueType>
any(ValueType&& value, typename boost::disable_if<boost::is_same<any&, ValueType> >::type* = 0)
: content(new holder< typename remove_reference<ValueType>::type >(static_cast<ValueType&&>(value)))
{
}
It checks for any& to disable itself on forwarding, but it doesn't
check for const any&. There are (non-template) constructors which take
const any&, but I guess some C++ rule favors the template
version. This results in infinite recursion of constructor invocations
between any() and holder().
Workaround is to make the lambda mutable, so that the argument type is
any& and template doesn't kick in.
Trace:
CU 0x4debed5, DIE 0x4f176e6>, this=0x602000044550) at /usr/include/boost/any.hpp:175
ql_query_test, CU 0x4debed5, DIE 0x4f6c509>, this=0x602000044518) at /usr/include/boost/any.hpp:72
CU 0x4debed5, DIE 0x4f176e6>, this=0x602000044510) at /usr/include/boost/any.hpp:175
ql_query_test, CU 0x4debed5, DIE 0x4f6c509>, this=0x6020000444e8) at /usr/include/boost/any.hpp:72
CU 0x4debed5, DIE 0x4f176e6>, this=0x6020000444e0) at /usr/include/boost/any.hpp:175
oost::any const>, void>::type*) (this=0x7ffff2ffa6f0, value=<unknown type in /home/tgrabiec/src/urchin2/build
/release/tests/urchin/cql_query_test, CU 0x4debed5, DIE 0x4f6c509>) at /usr/include/boost/any.hpp:72
5u> const&, std::vector<boost::any, std::allocator<boost::any> >, std::vector<boost::any, std::allocator<boos
t::any> >, basic_sstring<char, unsigned int, 15u> const&, boost::any)::{lambda(database&)#1}::operator()(data
base&) const::{lambda(std::unique_ptr<mutation_partition const, std::default_delete<mutation_partition> >)#1}
::unique_ptr(std::unique_ptr<mutation_partition const, std::default_delete<mutation_partition> >&&) (this=0x7
ffff2ffa6c0) at tests/urchin/cql_test_env.cc:126
Following std::async(), seastar::async(func) causes func to be executed
in a seastar thread, where it can block using future<>::get(). Whatever
func returns is converted to a future, and returned as async()s return
value.
Code calls failure_detector::is_alive on all cpus, so we start
failure_detector on all cpus. However, the internal data of failure_detector
is modified on cpu zero and it is not replicated to non-zero cpus.
This is fine since the user of failure_detector (the gossiper) accesses
it on cpu0 only.
This patch replaces the sstable read APIs from having "push" style,
to having "pull style".
The sstable read code has two APIs:
1. An API for sequentially consuming low-level sstable items - sstable
row's beginning and end, cells, tombstones, etc.
2. An API for sequentially consuming entire sstable rows in our "mutation"
format.
Before this patch, both APIs were in "push style": The user supplies
callback functions, and the sstable read code "pushes" to these functions
the desired items (low-level sstable parts, or whole mutations).
However, a push API is very inconvenient for users, like the query
processing code, or the compaction code, which both iterate over mutations.
Such code wants to control its own progression through the iteration -
the user prefers to "pull" the next mutation when it wants it; Moreover,
the user wants to *stop* pulling more mutations if it wants, without
worrying about various continuations that are still scheduled in the
background (the latter concern was especially problematic in the "push"
design).
The modified APIs are:
1. The functions for iterating over mutations, sstable::read_rows() et al.,
now return a "mutation_reader" object which can be used for iterating
over the mutation: mutation_reader::read() asks for the next mutation,
and returns a future to it (or an unassigned value on EOF).
You can see an example on how it is used in sstable_mutation_test.cc.
2. The functions for consuming low-level sstable items (row begin, cell,
etc.) are still partially push-style - the items are still fed into
the consume object - but consumpton now *stops* (instead of defering
and continuing later, as in the old code) when the consumer asks to.
The caller can resume the consumption later when it wishes to (in
this sense, this is a "pull" API, because the user asks for more
input when it wants to).
This patch does *not* remove input_stream's feature of a consumer
function returning a non-ready future. However, this feature is no longer
used anywhere in our code - the new sstable reader code stops the
consumption when old sstable reader code paused it temporarily with
a non-ready future.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
This change how the json formatter handle float and adds double support.
float and double conversion will throw exceptions when try to convert
inf or nan.
It also contains tests for both float and double including inf, -inf and
nan for both float and double.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Simplify the return value of map_difference() to return set of keys.
This makes it possible to use the function with value types such as
foreign_ptr<> that are non-copyable.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
