After commit 3ae81e68a0, we already support
in input_stream::consume() the possibility of the consumer blocking by
returning a future. But the code for sstable consumption had now way to
use this capability. This patch adds a future<> return code for
consume_row_end(), allowing the consumer to pause after reading each
sstable row (but not, currently, after each cell in the row).
We also need to use this capability in read_range_rows(), which wrongly
ignored the future<> returned by the "walker" function - now this future<>
is returned to the sstable reader, and causes it to pause reading until
the future is fulfilled.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
The current scheme of creating temporary files inside the current directory is
very fragile.
First, whenever we start writing new files, the test breaks if we forget to
include it in the list of removed files (likely).
Second, because we have to manually delete the files after we're done with
them, we can either forget, or if the test crashes, the final condition may not
run.
Instead of doing that, let's move the files to a separate directory. That makes
it a lot easier to make sure we're deleting all the relevant files, because we
can just wipe out the whole directory.
All tests are then wrapped in a function that makes sure that the proper setup
is in place.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
From Pekka:
"This series wires up migration manager column family announcements in
preparation for eventually storing column family schema in system
tables."
Convert code to use the deserialize() function and drop the duplicate
compose() wrapper that we inherited from Origin.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
If map value type is lw_shared_ptr<T>, for example, we need to be able
to pass our own value comparison function to the difference() function.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Use CQL for keyspace creation to make sure the keyspace definition also
exists in system tables.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Make sure keyspace exists for all test cases, not just ones that call
the create_table() helper.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
The parameters in the request holds the path parameters. The current
implementation based on unorder_map has a few downside:
1. Because path parameters are sometime used to mark a file path and
they can extend to multiple url section (may contain /) the value will
always contain the leading slash. Though this is true for files, for
the common case, it would be easier to return the value without it.
2. The []operator of the hash map is non const, this mean that it cannot
be used in the common case where the request object is passed as a const
reference.
3. There is no exists method in an ordered_map - Usually query
parameters are mandatory, still it is usefull to have an easy way of
testing if a parameter is found.
This series wrap the unordered_map implementation in a manner more
suitable for the request.
The [] operator will be const and retrun a copy of the parameter value
without the leading slash.
The at method will keep the old meaning. For clearer code, a path method
was added to indicate that the return value has a leading slash.
A set method is used for assignment.
Older code will continue to work without changes.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Store a lw_shared_ptr<keyspace_metadata> in struct keyspace so callers
in migration manager, for example, can look it up.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
This is my turn to make the same mistake as Nadav recently made: I have
written down the expected position from the test output itself, instead
of checking the file.
A position of 0x400c0000000000 is obviously ridiculous. The file itself is not
that big. My recent patch to fix the summary broke the test, and so we need to
fix it.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
This patch is built on the previous one, where access to members of sstable
class was made possible.
Index file is being properly generated, but it's important mentioning that
promoted indexes aren't supported yet.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
The function write_datafile was renamed to write_components and made a member
of sstable class because write of components requires access to private
members. This change is an important step towards the generation of components
other than data file.
The respective testcases were adapted to the changes.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
From Avi:
"This patchset prepares for adding sstables to the read path. Because sstables
involve I/O, their APIs return futures, which means that APIs that may call
those sstable APIs also need to return futures.
This patchset uses the two-space indent + do_with + reference aliases trick
to make patches more readable. Cleanup patches will follow once it is merged."
Pekka says:
"Clean up keyspace creation functions by using keyspace_metadata. While
at it, simplify creation by moving replication strategy initialization
to database.cc. This paves the way for adding keyspace metadata lookup
functionality to database that's needed by migration manager."
Our input_stream::consume() mechanism allows feeding data from an input
stream into a consumer, piece by piece, until the consumer doesn't want
any more. It currently assumed the input can block (when reading from disk),
but the consumption is assumed to be immediate. This patch adds support for
blocking in the consumption function: The consumer now returns a future
which it promises to fulfill after consuming the given buffer.
This patch goes further by somewhat simplifying (?) the interface of the
consumer. Instead of receiving a mysterious "done" lambda the consumer
is supposed to call when done (doesn't want any more input), the consumer
now returns a future<optional<temporary_buffer<char>>, which means:
1. The future is fulfilled when the consumer is done with this buffer
and either wants more - or wants to stop.
2. If the consumer wants to stop, it returns the *remaining* part of the
buffer it didn't want to process (this will be pushed back into the
input stream).
3. If the consumer is not done, and wants to consume more, it returns an
unset optional.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Initialize replication strategy when keyspace is being created now that
we have access to keyspace_metadata.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Use the keyspace_metadata type for keyspace creation functions. This is
needed to be able to have a mapping from keyspace name to keyspace
metadata for various call-sites.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Follow the naming convention set by user_types_metadata and rename
ks_meta_data to keyspace_metadata.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Pekka says:
"There's a user_types_metadata type defined in database.hh. Use that and
remove the left-over ut_meta_data from initial CQL translations."
Conflicts:
thrift/handler.cc
There's a user_types_metadata type in database.hh. Use it and drop the
left-over ut_meta_data from Origin.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
This patchset adds thread support to seastar. Threads can use the normal
seastar API, and may also "block" by calling future<>::get() on an
unavailable future. Synchronous and asynchronous code may be intermixed.
Threads may not issue blocking operating system calls.
Unfortunately, for some unknown reason, my compiler is not locally warning me
of signe comparison mismatches, but our builder's is.
We have a currently build failure that is hopefully fixed by this patch.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
The implementation of sleep() looks like a game of Seastar golf - doing
something in the minimum number of lines possible :-) Unfortunately, it
looks very clever, but not quite right. sleep() usually works correctly,
but the sanitizer (in the debug build) catches a use after free.
The problem was that we delete an object which contains a timer which
contains the callback (and std::function) - from inside this callback.
The workaround in this patch is to use our future chaining to only delete
the sleeper object after its future became ready - and at that point, none
of the sleeper object or code is needed any more.
This patch also includes a regression test for this issue. The test looks
silly (just sleeps and checks nothing), but in the debugging build it
failed (with a sanitizer reporting use-after-free) before this patch.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
