There can be multiple sends underway when first one detects an error and
destroys rpc client, but the rpc client is still in use by other sends.
Fix this by making rpc client pointer shared and hold on it for each
send operation.
Since we do not support shard to shard connections at the moment, ip
address should fully decide if a connection to a remote node exists or
not. messaging_service maintains connections to remote node using
std::unordered_map<shard_id, shard_info, shard_id::hash> _clients;
With this patch, we can possibly reduce number of tcp connections
between two nodes.
Instead of just destroying the client, stop() it first and wait for that
to complete. Otherwise any connect in progress, or any requests in progress,
will access a destroyed object (in practice it will fail earlier when
the _stopped promise is destroyed without having been fulfilled).
Depends on rpc client stop patch.
This patch introduce init.cc file which hosts all the initialization
code. The benefits are 1) we can share initialization code with tests
code. 2) all the service startup dependency / order code is in one
single place instead of everywhere.
It is used when a stream_transfer_task sends all the mutations inside
the messages::outgoing_file_message's it contains to notify the remote
peer this stream_transfer_task is done.
config.hh changes rapidly, so don't force lots of recompiles by including it.
Need to place seed_provider_type in namespace scope, so we can forward
declare it for that.
This patch serves as an example of how we can add wrappers for
ms.send_message and ms.register_handler.
When we convert all the users of them, we can make messaging_service.hh
do not include rpc.hh.
The messaging service is initialized very early, before we have the
proxy or query processor initialized. It is mostly fine, except for
the fact that the messaging service also finishes the initialization
of the storage service. That part will issue queries agains the system
tables, (as soon as we support them), and need to happen later.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
It is built on top of seastar rpc infrastructure. I've sorted out all
the message VERBs which Origin use. All of them can be implemented using
this messaging_service.
Each Verb contains a handler. There are two types of handlers, one
will return a message back to sender, the other will not. The former
can be registered using ms.register_handler(), the latter can be
registered using ms.register_handler_oneway().
Usage example:
To use messaging_service to send a message. All you need is:
messaging_service& ms = get_local_messaging_service();
1) To register a message hander:
ms.register_handler(messaging_verb::ECHO, [] (int x, long y) {
print("Server got echo msg = (%d, %ld) \n", x, y);
std::tuple<int, long> ret(x*x, y*y);
return make_ready_future<decltype(ret)>(std::move(ret));
});
ms.register_handler_oneway(messaging_verb::GOSSIP_SHUTDOWN, [] (empty_msg msg) {
print("Server got shutdown msg = %s\n", msg);
return messaging_service::no_wait();
});
2) To send a message:
using RetMsg = std::tuple<int, long>;
return ms.send_message<RetMsg>(messaging_verb::ECHO, id, msg1, msg2).then([] (RetMsg msg) {
print("Client sent echo got reply = (%d , %ld)\n", std::get<0>(msg), std::get<1>(msg));
return sleep(100ms).then([]{
return make_ready_future<>();
});
});
return ms.send_message_oneway<void>(messaging_verb::GOSSIP_SHUTDOWN, std::move(id), std::move(msg)).then([] () {
print("Client sent gossip_shutdown got reply = void\n");
return make_ready_future<>();
});
Tests:
send to cpu 0
$ ./message --server 127.0.0.1 --cpuid 0 --smp 2
send to cpu 1
$ ./message --server 127.0.0.1 --cpuid 1 --smp 2
