Unlike streaming in c*, scylla does not need to open tcp connections in
streaming service for both incoming and outgoing messages, seastar::rpc
does the work. There is no need for a standalone stream_init_message
message in the streaming negotiation stage, we can merge the
stream_init_message into stream_prepare_message.
"This series:
- Add more debug info to stream session
- Fail session if we fail to send COMPLETE_MESSAGE
- Handle message retry logic for verbs used by streaming
See commit log for details."
the serializer
This patch adds a specific template initialization so that the rpc would
use the serializer and deserializer that are auto-generated.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
When a verb timeout, if we resend the message again, the peer could receive
the message more than once. This would confuse the receiver. Currently, only
the streaming code use the retry logic.
- In case of rpc:timeout_error:
Instead of doing timeout in a relatively short time and resending a few
times, we make the timeout big enough and let the tcp to do the resend.
Thus, we can avoid resending the message more than once, of course, the
receiver would not receive the message more than once.
- In case of rpc::closed_error:
There are two cases:
1) Failing to establish a connection.
For instance, the peer is down. It is safe to resend since we know for
sure the receiver hasn't received the message yet.
2) The connection is established.
We can not figure out if the remote peer have received the message
already or not upon receiving the rpc::closed_error exception.
Currently, we still sleep & resend the message again, so the receiver
might receive the message more than once. We do not have better choice
in this case, if we want the resend to recover the sending error due to
temporary network issue, since failing the whole stream_session due to
failing to send a single message is not wise.
NOTE: If the duplicated message is received when the stream_session is done,
it will be ignored since it can not find the stream_manager anymore.
For message like, STREAM_MUTATION, it is ok to receive twice (we apply the
mutation twice).
TODO: For other messages which uses the retry logic, we need
to make sure it is ok to receive more than once.
While MUTATION and MUTATION_DONE are asynchronous by nature (when a MUTATION
completes, it sends a MUTATION_DONE message instead of responding
synchronously), we still want them to be synchronous at the server side
wrt. the RPC server itself. This is because RPC accounts for resources
consumed by the handler only while the handler is executing; if we return
immediately, and let the code execute asynchronously, RPC believes no
resources are consumed and can instantiate more handlers than the shard
has resources for.
Fix by changing the return type of the handlers to future<no_wait_type>
(from a plain no_wait_type), and making that future complete when local
processing is over.
Ref #596.
Message-Id: <1453048967-5286-1-git-send-email-avi@scylladb.com>
According to specification
(here https://wiki.apache.org/cassandra/InternodeEncryption)
when the internode encryption is set to `dc` the data passed between
DCs should be encrypted and similarly, when it's set to `rack`
the inter-rack traffic should encrypted.
Currently Scylla would encrypt the traffic inside a local DC in the
first case and inside the local RACK in the later one.
This patch fixes the encryption logic to follow the specification
above.
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1452501794-23232-1-git-send-email-vladz@cloudius-systems.com>
The verb belongs to a seaprate client to avoid potential deadlocks
should the throttling on connection level be introduced in the
future. Another reason is to reduce latency for version requests as it
can potentially block many requests.
This patch adds a new type of message, "REPAIR_CHECKSUM_RANGE" to scylla's
"messaging_service" RPC mechanism, for the use of repair:
With this message the repair's master host tells a slave host to calculate
the checksum of a column-family's partitions in a given token range, and
return that checksum.
The implementation of this message uses the checksum_range() function
defined in the previous patch.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
messaging_service will use private ip address automatically to connect a
peer node if possible. There is no need for the upper level like
streaming to worry about it. Drop it simplifies things a bit.
* Accept port + credentials + option for what to encrypt
* If set, enable a SSL listener at ssl_port
* Check outgoing connections by IP to determine if
they should go to SSL/normal endpoint
Requires seastar RPC patch
Note: currently, the connections created by messaging service
does _not_ do certificate name verification. While DNS lookup
is probably not that expensive here, I am not 100% sure it is
the desired behaviour.
Normal trust is however verified.
"When a node gain or regain responsibility for certain token ranges, streaming
will be performed, upon receiving of the stream data, the row cache
is invalidated for that range.
Refs #484."
"Fixes stream_session hangs:
1) if the sending node is gone, the receiving peer will wait forever
2) if the node which should send COMPLETE_MESSAGE to the peer node is gone,
the peer node will wait forever"
Get from address from cinfo. It is needed to figure out which stream
session this mutation is belonged to, since we need to update the keep
alive timer for this stream session.
Currently, if the node is actually down, although the streaming_timeout
is 10 seconds, the sending of the verb will return rpc_closed error
immediately, so we give up in 20 * 5 = 100 seconds. After this change,
we give up in 10 * 30 = 300 seconds at least, and 10 * (30 + 30) = 600
seconds at most.
It is oneway message at the moment. If a COMPLETE_MESSAGE is lost, no
one will close the session. The first step to fix the issue is to try to
retransmit the message.
When a node gain or regain responsibility for certain token ranges,
streaming will be performed, upon receiving of the stream data, the
row cache is invalidated for that range.
Refs #484.
