This patch replaces the sstable read APIs from having "push" style,
to having "pull style".
The sstable read code has two APIs:
1. An API for sequentially consuming low-level sstable items - sstable
row's beginning and end, cells, tombstones, etc.
2. An API for sequentially consuming entire sstable rows in our "mutation"
format.
Before this patch, both APIs were in "push style": The user supplies
callback functions, and the sstable read code "pushes" to these functions
the desired items (low-level sstable parts, or whole mutations).
However, a push API is very inconvenient for users, like the query
processing code, or the compaction code, which both iterate over mutations.
Such code wants to control its own progression through the iteration -
the user prefers to "pull" the next mutation when it wants it; Moreover,
the user wants to *stop* pulling more mutations if it wants, without
worrying about various continuations that are still scheduled in the
background (the latter concern was especially problematic in the "push"
design).
The modified APIs are:
1. The functions for iterating over mutations, sstable::read_rows() et al.,
now return a "mutation_reader" object which can be used for iterating
over the mutation: mutation_reader::read() asks for the next mutation,
and returns a future to it (or an unassigned value on EOF).
You can see an example on how it is used in sstable_mutation_test.cc.
2. The functions for consuming low-level sstable items (row begin, cell,
etc.) are still partially push-style - the items are still fed into
the consume object - but consumpton now *stops* (instead of defering
and continuing later, as in the old code) when the consumer asks to.
The caller can resume the consumption later when it wishes to (in
this sense, this is a "pull" API, because the user asks for more
input when it wants to).
This patch does *not* remove input_stream's feature of a consumer
function returning a non-ready future. However, this feature is no longer
used anywhere in our code - the new sstable reader code stops the
consumption when old sstable reader code paused it temporarily with
a non-ready future.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
This change how the json formatter handle float and adds double support.
float and double conversion will throw exceptions when try to convert
inf or nan.
It also contains tests for both float and double including inf, -inf and
nan for both float and double.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Simplify the return value of map_difference() to return set of keys.
This makes it possible to use the function with value types such as
foreign_ptr<> that are non-copyable.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Digest file stores adler checksum of data file converted into a
string. Testcase is added.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
CRC component is composed of chunk size, and a vector of checksums
for each chunk (at most chunk size bytes) composing the data file.
The implementation is about computing the checksum every time the
output stream of data file gets written. A write to output stream
may cross the chunk boundary, so that must be handled properly.
Note that CRC component will only be created if compression isn't
being used.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
After commit 3ae81e68a0, we already support
in input_stream::consume() the possibility of the consumer blocking by
returning a future. But the code for sstable consumption had now way to
use this capability. This patch adds a future<> return code for
consume_row_end(), allowing the consumer to pause after reading each
sstable row (but not, currently, after each cell in the row).
We also need to use this capability in read_range_rows(), which wrongly
ignored the future<> returned by the "walker" function - now this future<>
is returned to the sstable reader, and causes it to pause reading until
the future is fulfilled.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
The current scheme of creating temporary files inside the current directory is
very fragile.
First, whenever we start writing new files, the test breaks if we forget to
include it in the list of removed files (likely).
Second, because we have to manually delete the files after we're done with
them, we can either forget, or if the test crashes, the final condition may not
run.
Instead of doing that, let's move the files to a separate directory. That makes
it a lot easier to make sure we're deleting all the relevant files, because we
can just wipe out the whole directory.
All tests are then wrapped in a function that makes sure that the proper setup
is in place.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
From Pekka:
"This series wires up migration manager column family announcements in
preparation for eventually storing column family schema in system
tables."
Convert code to use the deserialize() function and drop the duplicate
compose() wrapper that we inherited from Origin.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
If map value type is lw_shared_ptr<T>, for example, we need to be able
to pass our own value comparison function to the difference() function.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Use CQL for keyspace creation to make sure the keyspace definition also
exists in system tables.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Make sure keyspace exists for all test cases, not just ones that call
the create_table() helper.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
The parameters in the request holds the path parameters. The current
implementation based on unorder_map has a few downside:
1. Because path parameters are sometime used to mark a file path and
they can extend to multiple url section (may contain /) the value will
always contain the leading slash. Though this is true for files, for
the common case, it would be easier to return the value without it.
2. The []operator of the hash map is non const, this mean that it cannot
be used in the common case where the request object is passed as a const
reference.
3. There is no exists method in an ordered_map - Usually query
parameters are mandatory, still it is usefull to have an easy way of
testing if a parameter is found.
This series wrap the unordered_map implementation in a manner more
suitable for the request.
The [] operator will be const and retrun a copy of the parameter value
without the leading slash.
The at method will keep the old meaning. For clearer code, a path method
was added to indicate that the return value has a leading slash.
A set method is used for assignment.
Older code will continue to work without changes.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Store a lw_shared_ptr<keyspace_metadata> in struct keyspace so callers
in migration manager, for example, can look it up.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
This is my turn to make the same mistake as Nadav recently made: I have
written down the expected position from the test output itself, instead
of checking the file.
A position of 0x400c0000000000 is obviously ridiculous. The file itself is not
that big. My recent patch to fix the summary broke the test, and so we need to
fix it.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
This patch is built on the previous one, where access to members of sstable
class was made possible.
Index file is being properly generated, but it's important mentioning that
promoted indexes aren't supported yet.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
The function write_datafile was renamed to write_components and made a member
of sstable class because write of components requires access to private
members. This change is an important step towards the generation of components
other than data file.
The respective testcases were adapted to the changes.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
From Avi:
"This patchset prepares for adding sstables to the read path. Because sstables
involve I/O, their APIs return futures, which means that APIs that may call
those sstable APIs also need to return futures.
This patchset uses the two-space indent + do_with + reference aliases trick
to make patches more readable. Cleanup patches will follow once it is merged."
Pekka says:
"Clean up keyspace creation functions by using keyspace_metadata. While
at it, simplify creation by moving replication strategy initialization
to database.cc. This paves the way for adding keyspace metadata lookup
functionality to database that's needed by migration manager."
Our input_stream::consume() mechanism allows feeding data from an input
stream into a consumer, piece by piece, until the consumer doesn't want
any more. It currently assumed the input can block (when reading from disk),
but the consumption is assumed to be immediate. This patch adds support for
blocking in the consumption function: The consumer now returns a future
which it promises to fulfill after consuming the given buffer.
This patch goes further by somewhat simplifying (?) the interface of the
consumer. Instead of receiving a mysterious "done" lambda the consumer
is supposed to call when done (doesn't want any more input), the consumer
now returns a future<optional<temporary_buffer<char>>, which means:
1. The future is fulfilled when the consumer is done with this buffer
and either wants more - or wants to stop.
2. If the consumer wants to stop, it returns the *remaining* part of the
buffer it didn't want to process (this will be pushed back into the
input stream).
3. If the consumer is not done, and wants to consume more, it returns an
unset optional.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Initialize replication strategy when keyspace is being created now that
we have access to keyspace_metadata.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Use the keyspace_metadata type for keyspace creation functions. This is
needed to be able to have a mapping from keyspace name to keyspace
metadata for various call-sites.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Follow the naming convention set by user_types_metadata and rename
ks_meta_data to keyspace_metadata.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
