From Vlad:
"Currently database always created a SimpleSnitch and ignores the corresponding parameter
provided by the user. This series fixes this situation:
- Changes the snitch creation interface to comply the Java-like interface that
has already been used in a topology_strategy classes family.
- Fix all the places where a SimpleSnitch has been created ignoring the user configuration."
Fix keyspace strategy options to preserve key-value ordering by
switching to std::map. We need this to be able to store the map in
database as JSON because unordered maps can cause the schema merging
code to attempt a keyspace update, which we don't support, even though
the values did not change.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
- Forbid explicit snitch creation with constructor.
- Allow the creation of snitches only with locator::make_snitch() template
function.
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
New in v4:
- Make sure the snitch is stopped before it's destroyed when _snitch_is_ready
is returned in an exceptional state.
New in v2:
- Change snitch_ptr to be std::unique_ptr<i_endpoint_snitch>
- abstract_replication_strategy::create_replication_strategy(): explicitly
specify (template) types of create_object() parameters.
- Re-arrange the loop in marge_keyspaces() so that lambdas that depend on
"this" complete before there is a chance that "this" gets destroyed.
- create_keyspace(): Don't add a new keyspace if a keyspace with this name
already exists.
- i_endpoint_snitch: added a stop() virtual method
- Added a stop() pure virtual method.
- Added an enum class snitch_state and a _state member initialized to snitch_state::initializing,
added an assert() in a destructor requiring _state to become snitch_state::stopped,
which should be set when stop() is complete.
- rack_inferring_snitch: added a stop() method.
- simple_snitch: added a stop() method.
- Added stop() methods to abstract_replication_strategy and keyspace.
- Updated database::stop() to wait for all keyspaces in _keyspaces to stop.
The field compressor is about saying which compressor algorithm
must be used in compression of sstable data file.
This is a small step towards compressed sstable data file.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
For some reason, I added a fsync call when the file underlying the
stream gets truncated. That happens when flushing a file, which
size isn't aligned to the requested DMA buffer.
Instead, fsync should only be called when closing the stream, so this
patch changes the code to do that.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
When a promise that still tracks its future is destroyed, but after
future::get() or future::then() was called, the promise thinks it was
destroyed before generating a value, and fails an assertion.
Fix be detaching the promise from the future as soon as get() or then()
remove the value.
cql_query_test segfaults in debug mode due to infinite recursion (see
trace below). The problem started to appear after Avi's always-defer
change.
It looks like the following boost::any() constructor template is
instantiated when boost::any is copied into a lambda:
// Perfect forwarding of ValueType
template<typename ValueType>
any(ValueType&& value, typename boost::disable_if<boost::is_same<any&, ValueType> >::type* = 0)
: content(new holder< typename remove_reference<ValueType>::type >(static_cast<ValueType&&>(value)))
{
}
It checks for any& to disable itself on forwarding, but it doesn't
check for const any&. There are (non-template) constructors which take
const any&, but I guess some C++ rule favors the template
version. This results in infinite recursion of constructor invocations
between any() and holder().
Workaround is to make the lambda mutable, so that the argument type is
any& and template doesn't kick in.
Trace:
CU 0x4debed5, DIE 0x4f176e6>, this=0x602000044550) at /usr/include/boost/any.hpp:175
ql_query_test, CU 0x4debed5, DIE 0x4f6c509>, this=0x602000044518) at /usr/include/boost/any.hpp:72
CU 0x4debed5, DIE 0x4f176e6>, this=0x602000044510) at /usr/include/boost/any.hpp:175
ql_query_test, CU 0x4debed5, DIE 0x4f6c509>, this=0x6020000444e8) at /usr/include/boost/any.hpp:72
CU 0x4debed5, DIE 0x4f176e6>, this=0x6020000444e0) at /usr/include/boost/any.hpp:175
oost::any const>, void>::type*) (this=0x7ffff2ffa6f0, value=<unknown type in /home/tgrabiec/src/urchin2/build
/release/tests/urchin/cql_query_test, CU 0x4debed5, DIE 0x4f6c509>) at /usr/include/boost/any.hpp:72
5u> const&, std::vector<boost::any, std::allocator<boost::any> >, std::vector<boost::any, std::allocator<boos
t::any> >, basic_sstring<char, unsigned int, 15u> const&, boost::any)::{lambda(database&)#1}::operator()(data
base&) const::{lambda(std::unique_ptr<mutation_partition const, std::default_delete<mutation_partition> >)#1}
::unique_ptr(std::unique_ptr<mutation_partition const, std::default_delete<mutation_partition> >&&) (this=0x7
ffff2ffa6c0) at tests/urchin/cql_test_env.cc:126
Following std::async(), seastar::async(func) causes func to be executed
in a seastar thread, where it can block using future<>::get(). Whatever
func returns is converted to a future, and returned as async()s return
value.
Code calls failure_detector::is_alive on all cpus, so we start
failure_detector on all cpus. However, the internal data of failure_detector
is modified on cpu zero and it is not replicated to non-zero cpus.
This is fine since the user of failure_detector (the gossiper) accesses
it on cpu0 only.
This patch replaces the sstable read APIs from having "push" style,
to having "pull style".
The sstable read code has two APIs:
1. An API for sequentially consuming low-level sstable items - sstable
row's beginning and end, cells, tombstones, etc.
2. An API for sequentially consuming entire sstable rows in our "mutation"
format.
Before this patch, both APIs were in "push style": The user supplies
callback functions, and the sstable read code "pushes" to these functions
the desired items (low-level sstable parts, or whole mutations).
However, a push API is very inconvenient for users, like the query
processing code, or the compaction code, which both iterate over mutations.
Such code wants to control its own progression through the iteration -
the user prefers to "pull" the next mutation when it wants it; Moreover,
the user wants to *stop* pulling more mutations if it wants, without
worrying about various continuations that are still scheduled in the
background (the latter concern was especially problematic in the "push"
design).
The modified APIs are:
1. The functions for iterating over mutations, sstable::read_rows() et al.,
now return a "mutation_reader" object which can be used for iterating
over the mutation: mutation_reader::read() asks for the next mutation,
and returns a future to it (or an unassigned value on EOF).
You can see an example on how it is used in sstable_mutation_test.cc.
2. The functions for consuming low-level sstable items (row begin, cell,
etc.) are still partially push-style - the items are still fed into
the consume object - but consumpton now *stops* (instead of defering
and continuing later, as in the old code) when the consumer asks to.
The caller can resume the consumption later when it wishes to (in
this sense, this is a "pull" API, because the user asks for more
input when it wants to).
This patch does *not* remove input_stream's feature of a consumer
function returning a non-ready future. However, this feature is no longer
used anywhere in our code - the new sstable reader code stops the
consumption when old sstable reader code paused it temporarily with
a non-ready future.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
This change how the json formatter handle float and adds double support.
float and double conversion will throw exceptions when try to convert
inf or nan.
It also contains tests for both float and double including inf, -inf and
nan for both float and double.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Simplify the return value of map_difference() to return set of keys.
This makes it possible to use the function with value types such as
foreign_ptr<> that are non-copyable.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Digest file stores adler checksum of data file converted into a
string. Testcase is added.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
CRC component is composed of chunk size, and a vector of checksums
for each chunk (at most chunk size bytes) composing the data file.
The implementation is about computing the checksum every time the
output stream of data file gets written. A write to output stream
may cross the chunk boundary, so that must be handled properly.
Note that CRC component will only be created if compression isn't
being used.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
After commit 3ae81e68a0, we already support
in input_stream::consume() the possibility of the consumer blocking by
returning a future. But the code for sstable consumption had now way to
use this capability. This patch adds a future<> return code for
consume_row_end(), allowing the consumer to pause after reading each
sstable row (but not, currently, after each cell in the row).
We also need to use this capability in read_range_rows(), which wrongly
ignored the future<> returned by the "walker" function - now this future<>
is returned to the sstable reader, and causes it to pause reading until
the future is fulfilled.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
The current scheme of creating temporary files inside the current directory is
very fragile.
First, whenever we start writing new files, the test breaks if we forget to
include it in the list of removed files (likely).
Second, because we have to manually delete the files after we're done with
them, we can either forget, or if the test crashes, the final condition may not
run.
Instead of doing that, let's move the files to a separate directory. That makes
it a lot easier to make sure we're deleting all the relevant files, because we
can just wipe out the whole directory.
All tests are then wrapped in a function that makes sure that the proper setup
is in place.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
From Pekka:
"This series wires up migration manager column family announcements in
preparation for eventually storing column family schema in system
tables."
Convert code to use the deserialize() function and drop the duplicate
compose() wrapper that we inherited from Origin.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
If map value type is lw_shared_ptr<T>, for example, we need to be able
to pass our own value comparison function to the difference() function.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Use CQL for keyspace creation to make sure the keyspace definition also
exists in system tables.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Make sure keyspace exists for all test cases, not just ones that call
the create_table() helper.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
The parameters in the request holds the path parameters. The current
implementation based on unorder_map has a few downside:
1. Because path parameters are sometime used to mark a file path and
they can extend to multiple url section (may contain /) the value will
always contain the leading slash. Though this is true for files, for
the common case, it would be easier to return the value without it.
2. The []operator of the hash map is non const, this mean that it cannot
be used in the common case where the request object is passed as a const
reference.
3. There is no exists method in an ordered_map - Usually query
parameters are mandatory, still it is usefull to have an easy way of
testing if a parameter is found.
This series wrap the unordered_map implementation in a manner more
suitable for the request.
The [] operator will be const and retrun a copy of the parameter value
without the leading slash.
The at method will keep the old meaning. For clearer code, a path method
was added to indicate that the return value has a leading slash.
A set method is used for assignment.
Older code will continue to work without changes.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
