result_set_builder's API is:
new_row
add
add
add
new_row
add
add
add
new_row
add
add
add
build
Since there is no end_row, it relies on an internal flag to see (in new_row
and in build) whether we need to end a previous row. The problem is that
we check if the row is empty(), which is true both for the first row, and
for an empty row (if add() is never called, e.g. "SELECT COUNT(*) FROM tab".
Fix by using optional<> to mark whether the row exists (new_row has been
called). This is ugly, but matches origin. We should improve that by
adding an explicit end_row().
This do a clean up in the sstring replace method, it also uses a
std::move for optimization.
The missing cbeging and cend method were edded to return a
const_iterator
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
To support HEAD version of DPDK, we need -mavx/-mavx2 on Seastar CFLAGS.
But we cannot enable it until Host CPU has the feature, so we need to check it.
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
OSv does not able to load separated library version of DPDK, so it's better to use combined library (libintel_dpdk.so).
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Test the sstables::data_consume_row() function by consuming a row from
an sstable with known contents, and comparing the expected row key,
deletion time, and for each cell - its name, value and timestamp.
The test tests this on both a compressed and non-compressed sstable
(the contents of the two sstables is the same, except the timestamp
which is different).
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
HTTP server improvements, from Amnon:
"This series continue the http server OSv code migration and adaptation.
It adds the ability to dynamically register a swagger definition file. It also
simplified the server creation and configuration.
By applying the seriese, the api-doc json file will be available at
http://localhost:10000/api-doc/
and the demo API will be available at:
http://localhost:10000/api-doc/demo/
The series also adds some missing functionality to sstring to support the boost
replace functionality."
[v2: rebase, move "&", and tone down recommendation of template lambda]
Before commit f49c065cd1, we had the buggy
but simple-looking code
return data_stream_at(pos).read_exactly(len);
This was a bug because we need to ensure that the temporary object returned
by data_stream_at(pos) continues to live until the read_exactly() future
concludes. We solved this bug by doing this unsightly/unreadable code:
auto stream = std::make_unique<input_stream<char>>(data_stream_at(pos));
auto fut = stream->read_exactly(len);
return fut.then([stream = std::move(stream)]
(temporary_buffer<char> buf) { return buf; });
Instead, we can use the new do_with() idiom, which was exactly designed
to make a temporary object live until a future concludes. So we can write
the much shorter, and easier to understand, code:
return do_with(data_stream_at(pos), [len] (auto &stream) {
return stream.read_exactly(len);
});
Note the C++14 template lambda (the "auto" in the argument of the lamda):
This lambda gets whatever we feed it (in this case, the stream returned
by data_stream_at). The "&" after the "auto" is important: without it,
the compiler tries to pass the object by value, which is impossible
(because it is not copyable).
Of course it would have also been possible to specify the stream's type
explicitly instead of using template lambda:
return do_with(data_stream_at(pos), [len] (input_stream<char> &stream) {
return stream.read_exactly(len);
});
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
This is a migration of the api_docs from OSv. It replaces the static
api-doc.json with a dynamic generated reply, this allows to register API
in run time.
The api_registry_builder is a helper class that holds the file and api
path, simplifying registring both the api_doc handler and registering
additional API.
To use the api_doc, first generate a api_registry_builder.
The registry supply two functions, one for registring the api_doc
handler and one for registering an API.
Both function are passed as an argument for the set_routes method of the
http_server_control object.
To find the handler, the get_exact_match in the routes object was needed
to become public.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
There are multiple tasks that need to be performed when setting and
running an http server, specifically the routes need to be set and the
server need to be start.
The server control wrap the server initiation functionality and adds an
iteration function for all routes.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Files transformers are used to replace file content before retrieving
it. Specifically, it is used to return swagger definition files with the
host and protocol replaced.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Now that http is a library, different internal apps would need to use it
and include its dependency list. To simplify that, the http general
dependencies where moved to a variable.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
The boost::replace_all uses insert and erase to perform a change, those
method are missing from sstring.
Following the std:string implemntation the implementation of both
functionalities is based on the replace method, either as c string
replace or with templated iterators.
Not all the variation of insert, replace and erase where added, but it
will be possible to add them in the future if needed by calling the
existing functionality.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
This code adds the ability to write statistics to disk.
On-disk format:
uint32_t Size;
struct {
uint32_t metadata_type;
uint32_t offset; /* offset into this file */
} metadata_metadata[Size];
* each metadata_metadata entry corresponds to a metadata
stored in the file.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Change function argument types and return values to bytes_opt.
Note: a comment in the code says that NULL arguments are not supported,
but it seems prudent to prepare for the day they will be.
Function handler in general and json function in particular are the
easiest way to add logic to to handler.
While in some cases the method can return immediately, there are cases
when it is required to perform an async operation, to support those
cases, the function handler was modified to use future.
If it receives an old style function, it would wrap the results in a
make_ready_future. This you could still assign a function like:
new function_handler([](const_req req) {
return "hello";
});
It would no also support a function that return a future json so it is
no possible to assign logic like:
new function_handler([](std::unique_ptr<request> req) {
return make_ready_future<json::json_return_type>("json-future");
});
For the future case note that auto-boxing still works, although you now
need to use make_ready_future.
The json_path was also modified to accept the new kind of function, to
support the common case of route definition based on the code
generation.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Add a function sstables::data_consume_row() which reads an entire row
(or several consecutive rows) at a given byte range in the data file,
and feeds them into a "row_consumer" implementation which the user provides.
The row_consumer's method consume_row_start() method is called at the
beginning of the (or each) row with its key and deletion information,
then the consume_cell() method is called for each of the row's cells,
and after all cells of the row, consume_row_end() is called.
The current implementation only supports regular cells, and not other
special cases like range tombstones and counters (see
https://github.com/cloudius-systems/urchin/wiki/SSTables%20Data%20File)
as I did not yet have sstables to test those on; The current
implementation will abort upon seeing these unsupported features.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
The solution we have in tree now in for testing is obviously superior than this.
Let's switch to that.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
We are currently testing internal members of the sstable by specifying a bunch of
friend classes in the sstable structure. We have established that this is not the
ideal solution, but it is working.
My proposal here is to change that slightly: have a placeholder class defined in
sstables.hh, that will then re-export publicly every method it wants to use. (Thanks
Avi for suggesting that)
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
This patch converts (for very small value of 'converts') some
replication related classes. Only static topology is supported (it is
created in keyspace::create_replication_strategy()). During mutation
no replication is done, since messaging service is not ready yet,
only endpoints are calculated.
Inspired in Gleb's previous patch, this patch adds a hash and comparison
operator for dht::token.
The previous patch, however, had a number of problems. Comparisons were failing
in tokens that were verified (by me) to be equal to the ones Origin was
generating.
The main reasons for that, was that the byte-comparison loop must be unsigned,
not signed.
With the above change, the comparison function would always succeed *except*
when the integer version of _data was that of a signed one.
Looking at Origin, one verifies that the Murmur3Partitioner class overrides the
comparison functions, and just does a Long comparison with the token.
This patch implements a similar mechanism. With that, a list of tokens
generated by origin in ascending order is verified by us to also be in
ascending order.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
From Avi:
This patchsets completes the conversion of scalar functions (TOKEN is still
missing, and maybe others, but the infrastructure is there).
Conflicts:
database.cc
Add a convenient function do_with(rvalue, func) which ensures that a
(moved copy of) rvalue will live until the future returned by func
concludes, and that func is passed this object.
Needing to do this is a recurring idiom in Seastar applications.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
[avi: mark inline]
+ if (from >= std::numeric_limits<T>::max()) {
Avi explains an issue with the snippet above from the function:
This misses the case where either type is signed. At best you'd
get a compiler warning about comparing types with different
signedness, at worst a negative value can be truncated.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
At present, no overloads are needed because collections are interned, and
all other implemented data types are singletons. Tuples and user defined
types will need an overload.
